Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Paraloslistillos
Paraloslistillos
Loading in …3
×
1 of 17

Agile Testing for Embedded and IoT Software Development

1

Share

Download to read offline

Much of the success of agile adoptions is due to the automated testing approach used in agile projects. Because many of these techniques were pioneered in the development of web applications, it can be difficult to see how these techniques can be leveraged for a project where software is being built for an embedded or Internet of Things (IoT) application. Thomas Stiehm describes ways to leverage agile testing techniques for embedded systems. Whether you are building a medical device, embedded controller, or IoT device, learn how to leverage these testing practices to create fully automated tests that fit into a DevOps build pipeline and help your team create higher-quality, more reliable software. Test automation, the best way to maintain and execute a comprehensive suite of regression tests, allows you to maintain control of your testing process while increasing test coverage. Join Thomas to see how you can take control of your test process by stepping up your test automation to the next level.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Agile Testing for Embedded and IoT Software Development

  1. 1.         W16   Agile  Testing   10/5/16  15:00             Agile  Testing  for  Embedded  and  IoT   Software  Development   Presented  by:         Thomas  Stiehm       Coveros,  Inc.     Brought  to  you  by:                 350  Corporate  Way,  Suite  400,  Orange  Park,  FL  32073     888-­‐-­‐-­‐268-­‐-­‐-­‐8770  ·∙·∙  904-­‐-­‐-­‐278-­‐-­‐-­‐0524  -­‐  info@techwell.com  -­‐  http://www.starwest.techwell.com/            
  2. 2.     Thomas  Stiehm       Thomas  Stiehm  has  been  developing  applications  and  managing  software   development  teams  for  eighteen  years.  As  CTO  of  Coveros,  he  is  responsible  for  the   oversight  of  all  technical  projects  and  integrating  new  technologies  and  application   security  practices  into  software  development  projects.  Most  recently,  Thomas  has   been  focusing  on  how  to  incorporate  DevOps  best  practices  into  distributed  agile   development  projects  using  cloud-­‐based  solutions  and  how  to  achieve  a  balance   between  team  productivity  and  cost  while  mitigating  project  risks.  Previously,  as  a   managing  architect  involved  in  agile  development  at  Digital  Focus,  Thomas  found   that  agile  is  the  only  development  methodology  that  makes  the  business  reality  of   constant  change  central  to  the  development  process.    
  3. 3. 9/21/16 1 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1 Agility. Security. Delivered. Agile Tes)ng for Embedded So3ware Development © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 2 About Coveros •  Coveros builds security-criIcal applicaIons using agile methods. •  Coveros Services •  Agile transformaIons •  Agile development and tesIng •  DevOps and conInuous integraIon •  ApplicaIon security analysis •  Agile & Security training •  Government qualificaIons •  DCAA approved rates and accounIng •  TS facility clearance Areas of Exper8se
  4. 4. 9/21/16 2 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 3 What is embedded so3ware •  Embedded soTware is soTware that controls a device •  A device is a combinaIon of soTware, firmware and hardware that make up a system •  The system has hardware components that are either sensors or manipulators •  The soTware and firmware control the hardware components, using the sensors to discover and the manipulators to carry out acIons •  Examples include: •  Medical devices •  Self driving cars •  Fitness wearables •  HVAC systems © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 4 What is agile tes)ng •  Agile tesIng is the the set of pracIces used to build quality in and make sure that soTware created by the team is of high quality •  The whole team is responsible for quality •  Test automaIon is integrated into the ConInuous IntegraIon and ConInuous Delivery pipeline •  TesIng of new funcIonality is conducted during the same sprint or iteraIon in which the funcIonality is developed •  Unit tesIng is used as part of the quality assurance process •  Acceptance tesIng is conducted using acceptance criteria created directly from the requirements
  5. 5. 9/21/16 3 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 5 User Acceptance Tes)ng (UAT) •  UAT is the process of ge]ng stakeholders to write down what results they want to see from the soTware in order to determine it works as the requirements intended •  Acceptance criteria are wri^en before development of the soTware begins •  Acceptance criteria are a form of requirements that are used to let the team know what key aspects of the requirements need to be met in order for the stakeholders to feel that they can accept the soTware increment •  TesIng of acceptance criteria happens at the end of an increment and gives the team feedback on how well they have created soTware that the stakeholders expected. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 6 Behavior Driven Development (BDD) •  BDD is a test automaIon process for tesIng soTware based on the requirements •  The tests are wri^en before the code is wri^en or at least independently of the code •  For some teams BDD is used to automate UAT tests •  Many BDD tools exist such as Cucumber and Behave •  Most tools use the Gherkin language, which uses the following format: •  Given ini8al condi8on •  When some ac8on or trigger •  Then resul8ng ac8on
  6. 6. 9/21/16 4 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 7 Test Driven Development (TDD) •  TDD, also called Test First Development, is the process of using unit tests to help design the code for an applicaIon •  The developer will write a test of what they intend the code to do and then write the code to make the test pass •  Red, Green, Refactor cycle •  This has a number of posiIve results: •  Makes the code more testable •  Makes the code more modular or self contained •  Makes the code more maintainable •  Gives the code a regression test suite •  Makes the intent of the code easier to understand © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 8 Con)nuous Integra)on (CI) •  CI is the process of integraIng and tesIng code upon check in •  In CI, integraIon is the process of integraIng a developer’s code with the code already in the repository •  The goal of CI is to make sure the code can build and pass an iniIal set of regression tests before other tesIng and quality assurance is applied to a specific build •  CI uses a build server such as Jenkins to do a clean build of the code and run a suite of unit tests •  If the build breaks, the people that checked in since the last successful build are informed and they are expected to get the build back to a working state
  7. 7. 9/21/16 5 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 9 Con)nuous Delivery (CD) •  CD is the process of taking a build through a build pipeline that consists of a CI process followed by several quality gates that verify the code through various quality checks including: •  Automated funcIonal tests •  StaIc code analysis •  Automated non-funcIonal tests like performance and security •  Automated deployment into progressively higher environments •  It could include on demand deployment to a manual test environment •  On demand deployment to a UAT environment •  On demand deployment to a demo environment •  ConInuous Deployment is automaIcally progressing to producIon including automaIcally deploying into producIon © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 10 Value Stream Mapping •  Value stream mapping is the process of determining how a product is created in your environment from incepIon to deployment and use in producIon •  By creaIng a value stream map you can figure out how many steps you have in your delivery process and how long each step takes •  Once you have a map of your process you can start to opImize and automate the different steps to streamline your process •  The goal is to reduce the Ime it takes to move requirements through your process and get new features into producIon
  8. 8. 9/21/16 6 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 11 Build Pipeline •  The build pipeline is the part of your value stream that takes checked in code and makes it into high quality, tested soTware that is either in producIon or ready to be distributed to customers •  CI/CD are a subset of your build pipeline •  AutomaIon, including test automaIon, is the only way to make a build pipeline work in a fast and efficient manner •  Many of the quality gates in a build pipeline are tesIng focused and require strong tesIng acumen in order to help deliver high quality soTware © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 12 Version Control •  Version control is the process of managing the code over Ime by allowing a project to keep track of the state of the code as it progresses •  All code, including tes6ng code and test automa6on code needs to be checked into version control •  ApplicaIon code and tesIng code should be versioned together since the test code acts on the applicaIon and changes over Ime to keep in sync with changes to the applicaIon code •  Version control tools include: •  Subversion •  Git •  Others
  9. 9. 9/21/16 7 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 13 Configura)on Management •  ConfiguraIon Management are the tools and process used to determine how soTware is setup in any given environment and how to account for all of the different files needed to make the soTware work •  This includes binaries, configuraIon files and environment se]ngs •  Most build pipelines have a component that manages configuraIon elements for each deployment so that the team can know exactly what is deployed and tested for every environment © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 14 Tes)ng So3ware vs. Hardware vs. System •  With embedded systems you are tesIng three things: •  SoTware – The code that manages the system •  Hardware – The mechanical and/or electrical parts of the system •  System – The combinaIon of hardware and soTware •  SomeImes firmware is tested like soTware, someImes it is tested like hardware (depending on how firm it is) •  IsolaIng what you are tesIng can help you create test automaIon that covers a much of the product as possible •  Some amount of manual tesIng might be unavoidable
  10. 10. 9/21/16 8 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 15 So3ware Tes)ng •  During soTware tesIng you are tesIng only the soTware part of your system •  This means that all input (sensor data) should be supplied by a known source, if possible by recording the data and playing it back, or simulaIng it •  Output can be ignored or simulated •  Record and playback, or simulaIon can be less expensive and easier to setup •  You should be focused on tesIng the business logic of the soTware, i.e. the calculaIons and determinaIons that soTware makes •  You can also focus on tesIng the user interface or interacIve elements of the system © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 16 Hardware Tes)ng •  Hardware tesIng can be agile as well •  Hardware can have a different cadence than soTware •  Agile hardware tesIng is about automaIng as much of the tesIng process as possible •  SoTware tools, both homegrown and third party, can be used to automate hardware tesIng •  Using the system soTware the controls that hardware to test the hardware is acceptable but keep the focus on the hardware
  11. 11. 9/21/16 9 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 17 System Tes)ng •  System tesIng is the combinaIon of the soTware and hardware working together •  Depending on the nature of your embedded system, this can be hard to automated or automate completely •  Depending on the nature of your embedded system there may be a good deal of tooling that can help you test the system or there may be nothing at all •  TesIng the system is criIcal because you don’t know how the soTware and hardware will interact unIl you actually have them interact © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 18 Test Automa)on vs. Manual Tes)ng •  Agile tesIng focuses on test automaIon and in some types of applicaIons it is possible to automate nearly all of the tesIng •  With the focus on test automaton it is easy to think that manually tesIng has no place in agile but that isn’t true •  Manual tesIng is oTen a precursor to funcIonal test automaIon •  Manual tesIng can be cheaper or more effecIve for some features of an embedded system •  Exploratory tesIng can exercise the system in a way that test automaIon will not and find deeper issues
  12. 12. 9/21/16 10 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 19 How much can you automate? •  Look at this from a number of points of view: •  What can you afford •  What do you have the skills or abiliIes to automate •  What is possible •  What is the most beneficial •  Finding a balance between test automaIon and manual tesIng will be a challenge that all projects face •  The answer will change over Ime as you learn and as your code improves © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 20 Coding Standards (for tests too) •  Using coding standards means: •  Making the coding standards •  Enforcing coding standards using code reviews •  It doesn’t ma^er what specific coding standards you use so much as you have them and the team agrees to them •  If possible, use an exisIng coding standard for your technology •  Make coding standards compliance part of your code review •  Be^er yet, use a code forma^er to apply the standard during code commit •  Apply coding standards to your tests as well
  13. 13. 9/21/16 11 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 21 Code Inspec)ons •  Code inspecIons include a number of things: •  Code reviews •  StaIc code analysis •  Secure code reviews •  Format and style reviews •  Automate what you can, format and style are easy to automate •  StaIc code analysis can be a good input into code reviews •  Code review tools can help, things like Crucible and ReviewBoard •  A secure code reviews is different than a code review and is a different skill set, you should master both © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 22 Unit Tes)ng •  Unit tests are developer tests that are used to make sure that the code that they write behaves as the developer intended •  Unit tests are wri^en in the language and technology planorm of the applicaIon code and oTen use a framework like Unity (see h^p:// www.throwtheswitch.org/unity/) •  Unit tests might be run on the dev planorm or the target planorm, this oTen depends on how the tooling works •  Sample code: void test_FuncIonWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void) { TEST_ASSERT_EQUAL(0, FindFuncIon_WhichIsBroken(78)); TEST_ASSERT_EQUAL_HEX(0x5a5a, FuncIonWhichReturnsLocalVariable()); }
  14. 14. 9/21/16 12 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 23 BDD Sample Code Scenario: Subtract two numbers Given I have a calculator on When I enter ”25" into the calculator And I enter ”10" into the calculator And I press subtract Then the result should be "15" on the screen Scenario: Add two numbers Given I have a calculator on When I enter ”25" into the calculator And I enter ”45" into the calculator And I press add Then the result should be ”70" on the screen © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 24 Mock Objects •  Mock objects are replacements for the dependencies of the code under test •  They are used to make it easier to test the code by responding the same way all the Ime •  There are a number of good mock object frameworks including CMock (see h^p://www.throwtheswitch.org/cmock/)
  15. 15. 9/21/16 13 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 25 Test Fixtures •  Test fixtures are the code that creates a fixed state that is used to baseline tests so that the test environment is the same each Ime the tests are run •  Test frameworks do this by exposing methods or funcIons that allow for the configuraIon of dependent objects or for the creaIon of mock objects •  For a data related test a test fixture might load a specific set of data into the database •  Most unit tesIng frameworks and many test automaIon tools have test fixtures build into them © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 26 Working with Simulators •  A simulator is soTware or hardware used for tesIng that has the same interface as the applicaIon or system it is replacing •  Simulators provide a consistent interface for tesIng, much like test fixtures or mock objects •  Simulators are more full featured and can react to input, in more varied ways that other tesIng tools •  The idea is that a simulator will react more like the real component in the real world •  Simulators can be expensive to build and maintain but there may be no other opIon
  16. 16. 9/21/16 14 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 27 Tes)ng services for embedded so3ware •  Many Internet of Things (IoT) devices use services •  Services are Internet accessible bits of funcIonality that do things like: •  Collect data •  Calculate results •  Coordinate resources •  TesIng these services can be easier to automate then the devices themselves •  Scale can be an issue for services •  Network availability or reliability can be an issue for the devices •  Performance tesIng is oTen important for tesIng services © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 28 What to take away from the presenta)on •  Test automaIon is the key to agile tesIng •  Do test automaIon planning while you do test planning •  Create test suites that test your soTware separately from your hardware and another suite that tests the system as a whole
  17. 17. 9/21/16 15 © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 29 Ques)ons? •  Tom SIehm •  tom.sIehm@coveros.com •  @thomassIehm

×