A Multi-Dimensional Test Automation Framework for Embedded Wireless Platforms: Configuration Modeling and Regression Testing
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V1I1P110Keywords:
Embedded Wireless Platforms, Multi-Dimensional Test Automation, Configuration Modeling, Covering Arrays, Wireless Sensor Networks, Configuration Coverage, Regression TestingAbstract
Embedded wireless platforms combine hardware variability, firmware variability, radio-protocol variability, and physical environmental variability, and every fielded firmware release must be validated against a configuration space that grows combinatorially with each new configuration parameter. Although experiment execution has been automated through testbed research and regression test selection and combinatorial configuration sampling have been automated through software-engineering research, the two lines of research are not closely coupled: testbed studies do not explicitly model the configuration space, and regression-selection studies do not consider the physical execution cost of the hardware targets. The article brings together the evidence from twenty-two primary sources that cover wireless testbeds, temperature-controlled experimentation, combinatorial covering suites, software product line (SPL) test-configuration generation, and industrial regression testing to establish and test a Multi-Dimensional Test Automation Framework (MDTAF) for embedded wireless platforms. MDTAF models each platform release as a multi-dimensional configuration space and generates constraint-aware covering arrays for the space over hardware topology, features of the firmware, protocol stacks, security options, and emulated environmental profiles, and couples the resulting test configurations with a cost-aware regression orchestrator that combines historical fault, recency, source-change, and duration prioritizers. We formalize the configuration-space model, coverage metrics and regression-selection objective, and we report an evidence synthesis (not a new experiment): covering arrays of strength two reduce an 81-configuration RAID space to nine configurations; SPL test-relevance analysis reduces test-configuration spaces by up to 3,528 times in the reported benchmark cases; history-based prioritization results in detecting 30% more regression faults with 20% of the suite at 40% lower execution time; and a production prioritizer can shift two-thirds of the faults to the first third of the nightly suites. We also report verified replay accuracy (average error 0.12 – 0.18 °C) for environment emulation. When the evidence cited does not justify the full claim for the integrated framework, we explicitly acknowledge this limitation and lay out an evaluation plan that can be reproduced to test the hypotheses of the design, the proposed MDTAF, and that is grounded in evidence.
References
[1] S. Yoo and M. Harman, "Regression testing minimization, selection and prioritization: a survey," Software Testing, Verification and Reliability, vol. 22, no. 2, pp. 67–120, 2012 (published online 2010), doi: 10.1002/stvr.430.
[2] A. Hartman, "Software and hardware testing using combinatorial covering suites," in Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, M. C. Golumbic and I. B.-A. Hartman, Eds. New York, NY, USA: Springer, 2005, ch. 10, pp. 237–266, doi: 10.1007/0-387-25036-0_10.
[3] V. Garousi, M. Felderer, Ç. M. Karapıçak, and U. Yılmaz, "What we know about testing embedded software," IEEE Software, vol. 35, no. 4, pp. 62–69, Jul./Aug. 2018, doi: 10.1109/MS.2018.2801546.
[4] C. A. Boano, M. Zúñiga, J. Brown, U. Roedig, C. Keppitiyagama, and K. Römer, "TempLab: A testbed infrastructure to study the impact of temperature on wireless sensor networks," in Proc. 5th Int. Conf. on Networked Systems (NetSys), Paderborn, Germany, 2014, pp. 95–106, doi: 10.1109/NetSys.2014.6848297.
[5] S. Bouckaert, W. Vandenberghe, B. Jooris, I. Moerman, and P. Demeester, "The w-iLab.t testbed," in Testbeds and Research Infrastructures. Development of Networks and Communities (TridentCom 2010), LNICST 46. Berlin, Germany: Springer, 2011, pp. 145–154, doi: 10.1007/978-3-642-17851-1_11.
[6] J. Beutel, M. Dyer, R. Lim, C. Plessl, M. Wöhrle, M. Yücel, and L. Thiele, "Automated wireless sensor network testing," in Proc. 4th Int. Conf. on Networked Sensing Systems (INSS), Braunschweig, Germany, Jun. 2007, pp. 303–310, doi: 10.1109/INSS.2007.4297445.
[7] I. Minakov, R. Passerone, A. Rizzardi, and S. Sicari, "A comparative study of recent wireless sensor network simulators," ACM Trans. Sensor Networks, vol. 12, no. 3, art. 20, Jul. 2016, 39 pp., doi: 10.1145/2897511.
[8] A. Yushev, M. Schappacher, and A. Sikora, "Titan TTCN-3 based test framework for resource constrained systems," MATEC Web of Conf., vol. 75, art. 06005, 2016, doi: 10.1051/matecconf/20167506005.
[9] J. Grabowski, I. Schieferdecker, and A. Ulrich, "History, status, and recent trends of the testing and test control notation version 3 (TTCN-3)," Software Testing, Verification and Reliability, vol. 24, no. 4, pp. 215–225, 2014, doi: 10.1007/s10009-014-0302-9 (editorial, Int. J. Softw. Tools Technol. Transfer).
[10] J. Guan and J. Offutt, "A model-based testing technique for component-based real-time embedded systems," in Proc. IEEE 8th Int. Conf. on Software Testing, Verification and Validation Workshops (ICSTW), Graz, Austria, Apr. 2015, pp. 1–10, doi: 10.1109/ICSTW.2015.7107407.
[11] D. Marijan, A. Gotlieb, and S. Sen, "Test case prioritization for continuous regression testing: An industrial case study," in Proc. IEEE Int. Conf. on Software Maintenance (ICSM), Eindhoven, The Netherlands, Sep. 2013, pp. 540–543, doi: 10.1109/ICSM.2013.91.
[12] P. E. Strandberg, D. Sundmark, W. Afzal, T. J. Ostrand, and E. J. Weyuker, "Experience report: Automated system level regression test prioritization using multiple factors," in Proc. IEEE 27th Int. Symp. on Software Reliability Engineering (ISSRE), Ottawa, ON, Canada, Oct. 2016, pp. 12–23, doi: 10.1109/ISSRE.2016.23.
[13] C. Henard, M. Papadakis, M. Harman, Y. Jia, and Y. Le Traon, "Comparing white-box and black-box test prioritization," in Proc. 38th Int. Conf. on Software Engineering (ICSE), Austin, TX, USA, May 2016, pp. 523–534, doi: 10.1145/2884781.2884791.
[14] C. Henard, M. Papadakis, and Y. Le Traon, "Mutation-based generation of software product line test configurations," in Search-Based Software Engineering (SSBSE 2014), LNCS 8636. Cham, Switzerland: Springer, 2014, pp. 92–106, doi: 10.1007/978-3-319-09940-8_7.
[15] C. H. P. Kim, D. Batory, and S. Khurshid, "Reducing combinatorics in testing product lines," in Proc. 13th Int. Conf. on Generative Programming: Concepts and Experiences (GPCE), Västerås, Sweden, Sep. 2014, pp. 57–68, doi: 10.1145/2658761.2658767 (ACM 1960275.1960284).
[16] A. Hervieu, B. Baudry, and A. Gotlieb, "PACOGEN: Automatic generation of pairwise test configurations from feature models," in Proc. IEEE 22nd Int. Symp. on Software Reliability Engineering (ISSRE), Hiroshima, Japan, Nov./Dec. 2011, pp. 120–129, doi: 10.1109/ISSRE.2011.17.
[17] F. Ensan, E. Bagheri, and D. Gašević, "Evolutionary search-based test generation for software product line feature models," in Advanced Information Systems Engineering (CAiSE 2012), LNCS 7328. Berlin, Germany: Springer, 2012, pp. 633–650, doi: 10.1007/978-3-642-31095-9_40.
[18] M. B. Cohen, "Designing test suites for software interaction testing," Ph.D. dissertation, Dept. Comput. Sci., Univ. of Auckland, Auckland, New Zealand, 2004.
[19] R. C. Bryce, C. J. Colbourn, and M. B. Cohen, "A framework of greedy methods for constructing interaction test suites," in Proc. 27th Int. Conf. on Software Engineering (ICSE), St. Louis, MO, USA, May 2005, pp. 146–155, doi: 10.1145/1062455.1062495.
[20] R. C. Turban, C. J. Colbourn, and D. Hoskins, "Experimental designs in software engineering: D-optimal designs and covering arrays," in Proc. 2004 ACM Workshop on Interdisciplinary Software Engineering Research (WISER, FSE-12), Newport Beach, CA, USA, 2004, pp. 55–66, doi: 10.1145/1029997.1030009.
[21] J. Torres-Jimenez, H. Avila-George, and E. F. Parra-Gonzalez, "Binary test-suites using covering arrays," Int. J. Software Engineering and Knowledge Engineering, vol. 28, no. 9, pp. 1321–1337, 2018, doi: 10.1142/S0218194018500377.
[22] D. Benavides, A. Felfernig, J. A. Galindo, and F. Reinfrank, "Automated analysis in feature modelling and product configuration," in Software Reuse for Competitive Advantage (ICSR 2013), LNCS 7925. Berlin, Germany: Springer, 2013, pp. 160–175, doi: 10.1007/978-3-642-38977-1_11.
[23] C. Elsner, P. Ulbrich, D. Lohmann, and W. Schröder-Preikschat, "Consistent product line configuration across file type and product line boundaries," in Software Product Lines: Going Beyond (SPLC 2010), LNCS 6287. Berlin, Germany: Springer, 2010, pp. 151–165, doi: 10.1007/978-3-642-15579-6_13.
[24] S. She, R. Lotufo, T. Berger, A. Wasowski, and K. Czarnecki, "Reverse engineering feature models," in Proc. 33rd Int. Conf. on Software Engineering (ICSE), Honolulu, HI, USA, May 2011, pp. 461–470, doi: 10.1145/1985793.1985856.
[25] D. R. Kuhn, R. N. Kacker, and Y. Lei, Introduction to Combinatorial Testing. Boca Raton, FL, USA: CRC Press, 2013, ch. 12 ("Combinatorial coverage as an aspect of test quality" per the authors' related presentations), doi: 10.1201/b14957.
[26] L. Yu, Y. Lei, R. N. Kacker, and D. R. Kuhn, "ACTS: A combinatorial test generation tool," in Proc. IEEE 6th Int. Conf. on Software Testing, Verification and Validation (ICST), Luxembourg, Mar. 2013, pp. 370–375, doi: 10.1109/ICST.2013.57.
[27] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG: A general strategy for t-way software testing," in Proc. 14th IEEE Int. Conf. and Workshops on the Engineering of Computer-Based Systems (ECBS), Tucson, AZ, USA, Mar. 2007, pp. 549–556, doi: 10.1109/ECBS.2007.47.
[28] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, "The AETG system: An approach to testing based on combinatorial design," IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437–444, Jul. 1997, doi: 10.1109/32.595861.
