What is Smoke Testing ?

Smoke testing is non-exhaustive software testing, ascertaining that the most crucial functions of a program work, but not bothering with finer details. The term comes to software testing from a similarly basic type of hardware testing, in which the device passed the test if it didn’t catch fire the first time it was turned on. A daily build and smoke test is among industry best practices advocated by the IEEE (Institute of Electrical and Electronics Engineers).

The original version of smoke testing predates both hardware and software testing and is still used to test the integrity of a variety of systems by placing a smoke bomb inside some kind of a chamber to see if there are any leaks for the smoke to escape through.

Guidelines for Smoke Testing

Work with the Developer

Because smoke testing focuses on changed code, you must work with the developer who wrote the code. You will have to understand:

  • What changed in the code. To understand the change, you will also have to understand the technology used; the developer can help explain it.
  • How the change affects the functionality.
  • How the change affects the interdependencies of various components.

Conduct a Code Review Before Smoke Testing

Before you run a smoke test, conduct a code review that focuses on any changes in the code. Code reviews are the most effective and efficient method to validate code quality and ensure against code defects and faults of commission. Smoke tests ensure that the primary critical or weak area identified either by code review or risk assessment is primarily validated, because if it fails the testing cannot continue.

Install Private Binaries on a Clean Debug Build

Because a smoke test must focus on validating only the functional changes in updated binaries, the test must run on a clean test environment by using the debug binaries for the files being tested.

Create Daily Builds

Daily builds require the team members to work together and encourage the developers to stay in sync. If iterations of new builds are delayed, the delay can easily cause products with multiple dependencies to get out of sync. Following a process of building daily and smoke testing any changed or new binaries ensures high quality.

Web and Load Testing

When you build your Web tests and load tests, it is a good practice to run a smoke test before running any long, heavy test. In Web and in load testing, smoke testing is short, light test. You use a smoke test to validate that everything is correctly configured and running as expected before running your tests for performance or stress testing.

How to Implement Smoke Tests

To implement smoke tests, the quality assurance (QA) lead will develop a set of test cases that are run any time a new release is provided from the development team. These set of test cases are used to test the major functional areas of the system.  They are not nit-picky test cases that test low-level detail of the application, they are higher level test cases that test overall functionality.  Normally, a smoke test will consist of 20 to 30 high level test cases.

To ensure quality awareness, the smoke test cases are publicized to the development team ahead of time, so that they are aware of the quality expectations.  If a build fails a smoke test, it should not be taken lightly, as this may illustrate a discipline problem with the development team.  Each developer should be held responsible for ensuring that their code does not break the smoke test.  Under no circumstance should the QA team agree to continue testing the release until all the smoke test cases have passed.

Some fun ways to enforce smoke test quality is to hold developers to this quality standard by peer pressure.  For example, developers that break the smoke test may have their name posted on your department bulletin board in the “Hall of Shame” until the developer provides a good build.    Or you may consider taping a sucker to the door of the developer that broke the smoke test until the developer fixes the build.   Another good technique is to have the project manager allocate a budget for a team party upon the software being successfully implemented.  Then if teams break the smoke test, a fine is levied, decreasing the total budget for the team party with each infraction.

Typical characteristics of Smoke Testing:

  • It exercises the entire system from end-to-end.
  • It is not exhaustive but should be capable of exposing major problems.
  • It ensures that the major functionality is working and the build is stable enough for further testing thoroughly.

Advantages of Smoke Testing:

  • Reduced Integration Risk: Since smoke testing is carried out the integration problems are uncovered at a much earlier stage than late in the cycle.
  • Finds Major Problems: A good designed smoke test can increase the probability of finding a major problem when software is built early in the cycle. Thus you catch bugs earlier in the cycle.
  • Can save time and cost – If a major problem is detected at the stage when the software is ready built, it can save huge time and cost if the same error was discovered late in the cycle.

Features of Automation Test Suite

Ensure a Automation test suite that has the following features:

  • Re-usability   Develop a robust function library to be used across the projects and within the application.
  • Robustness – Develop a robust test suite that can be easily used for testing the application in different environments e.g. development, testing and production.
  • Portability – To ensure easy installation of the suite in different environments.
  • Modularity – Develop modular scripts to ensure maintainability, scalability and readability of the scripts.
  • Scalability – Scalable scripts which can be upgraded/ enhanced.
  • Maintainability – The approach should be such where in major effort goes in to the planning and developing the scripts where as least possible effort is required to maintain the same.
  • Data Driven Testing – To perform positive and negative tests using data driven scripts.

Checklist for Automation Framework Planning:

  • Identify the Scenarios / Test Cases for Smoke – 6 Identified Done
  • Execute the Scenarios / Test Cases Manually at least 5 times – Done
  • Get the Scenarios approved from QA Manager / Product Manager / Client – Done
  • Create the Folder Structure – Done, C:AST
  • Create the shared object repository – Done
  • Identify the Actions for the approved scenarios – Done with 8 Actions
  • Create test data (take inputs from the manual testing team / SME’s) – Done
  • Generate the Actions / Action library by hard coding the data – 8 reusable created, 0-pending
  • Separate the data from the Actions / Reusable – Done
  • Integrate / call the Actions in the main test – Done
  • Execute & debug the Actions / Reusable – Done
  • Get the Actions reviewed by the lead / manager / client – Done
  • Create the AOM – Done
  • Use Environment Variables – Done –Build in, user defined Functions
  • Schedule the Smoke Test – Done