Engineering Working Group/Scoring Criteria
General remarks
No weights for the following criteria have been decided on, and this is not a top priority. The qualitative feedback from the criteria is expected to be useful already on its own. Suitable weights are more likely to be a matter of fine-tuning over the time, and we may find that we prefer not to exactly quantify them.
Note that there is some overlap between the criteria, because things like automated tests or communications skills help in many different ways. Please note also that some hard criteria like the requirement to make the software Open Source do not need to be restated here because they are mandatory anyway for all bidders.
Meeting the requirements
Does the bid meet the requirements from the call for proposals? Given that we allow for questions from the bidder answered to all bidders, the final bid should always fulfill all of the requirements.
Part of the requirements are the general objectives of OpenStreetMap, the OSMF, and the EWG. The EWG will ensure that these objectives are contained in each call for proposal.
Documentation
When there are APIs to implement, are these APIs comprehensively documented? When there are user interfaces to implement, what is the i18n strategy? The intended observable data model shall be sketched already in the bid. We consider the efforts reserved for documentation, planned documentation formats, and strategies for user acceptance tests with the community.
The EWG reserves the right to reuse all or parts of the bid as documentation when community is dissatisfied with the delivered documentation.
Security
Does the bid rely on now or later vulnerable components? In the best case, the implemented software is self-contained. In the worst case, it relies, directly or indirectly on components known to be stale and have security holes. To depend on external components is often bad, but only slightly bad if the component is well maintained and can be reasonably be expected to remain well maintained in the future. It is also only slightly bad if the external component is simple enough to expose only an insignificant attack surface.
Bonus points for a well-understandable security model in the bid.
Test cases, test strategy, and testing
The functionality shall be covered by automated tests. Good testing is known to be hard. Hence we look in particular on the efforts reserved for the testing strategy, finding test cases, and setting up the automated tests. We also value whether the laid-out test strategy is easy to understand and comprehensive. Bids may contain examples that can also serve as test cases.
Maintainability
What lifetime proposes the bid for the solution? What effort on maintenance do we expect for that lifetime? Good is a self-contained solution with little source code in a well-established programming language. Also good is if other work from the applicant is well-structured and well-commented. Excellent is a comprehensive coverage with automated tests on the functional level. Code coverage is good, but we know that that metric can be gamed. Bad is deeply tied dependencies, particular when it is difficult or impossible to find a drop-in replacement for that dependency.
Interoperability
Are the proposed interfaces accessible with state-of-the-art protocols? Open formats are a must-have. Simpler formats are better. E.g. JSON beats XML because XML allows for more complexity. Commonly used formats are better than arcane ones.
Cost
If the cost a for bid are unusally high then we may choose a slightly less complete but more economical offer. We may exclude bids with significantly lower cost if there is no good explanation why this bid is cheaper, that we do not trust the bidder to be able to fulfill the task.
Communications with the community
The bidders are expected to conduct the communication with the community. There should be a plan for with communication channels will get which information, although it is understood that there might and shall be always more communication if helpful.
The EWG reserves the non-exclusive right to publish the successful bid after removing personal data of the bidder or bidders. The EWG may also publish only parts of the successful bid.
Person
We encourage the bidders to tell about the person or people that actually do the work. Experience is an advantage when it comes to experience with the community and their conventions and priorities. Likewise, programming experience will give the bidder's estimations more credibility.
Project management
Although we expect bidders to be able to work self-sustained and even to incorporate dynamic feedback from the community without moderation, the EWG would like to see a project plan such that the EWG can at any given time give a reasonable exact estimation to third parties how much progress has already been made.