{
"$type": "site.standard.document",
"bskyPostRef": {
"cid": "bafyreibq7nx7fknuqwwxniw72hx4uqpjiro3gmb3tqsi2snyjxbeutb6mq",
"uri": "at://did:plc:i2ne3m5q6oq4jcnvn4k55skm/app.bsky.feed.post/3mnybrubgy732"
},
"coverImage": {
"$type": "blob",
"ref": {
"$link": "bafkreicz3epjoxuag7uanlxkumf5qobdfjmfs5qe65eyqnwh6sn2turbea"
},
"mimeType": "image/png",
"size": 1023881
},
"description": "Contract work is occasionally... fun... more so when it involves building labs.",
"path": "/lets-build-a-lab-part-2/",
"publishedAt": "2026-06-11T03:26:36.000Z",
"site": "https://prose.winterschon.com",
"tags": [
"Follow the Sun",
"standards are being used, implemented, tested, and validated",
"use SST and parallelization correctly",
"Peter principle"
],
"textContent": "So many words. Even today with my own writing I want a TL;DR, so fuck it here's a not-boilerplate version of one standardized **_\"Statement of Work\"_** contracts that is oriented on building labs. Timeline are negotiable, scope is negotiable, hourly rate is not often negotiable - I learned that lesson a long time ago.\n\nFor those not familiar, the **SoW** is a consulting agreement between contractor and employer, which defines the scope and timeline and requirements etc to _\"do the needful\"_ in a broader sense of everyone's favorite phrase from the 'ol \"Follow the Sun\" SLA support model from the early pre-Cloud era _(sorry cloudbabies it's from when you were single digits old or not yet born)_.\n\nThis specific SoW has been used in prior endeavors which eventually follow the typical enterprise hardware lifecycle in no specific order [1]: forklifts, retros, rebuilds, expansion, and... words.. so many words.\n\nLabs last a long time, so it's important to build things correctly the first time - not waste time hacking together whatever compromised decision trees were necessary due to last-minute planning (or no planning at all) and little to no experience with building labs. Remember, if you are a specialized engineer who excels in one or two areas, that does not imply domain specific knowledge from adjacent or entirely different sides of engineering worlds. I'm a systems architect .. I don't build jet engines (though it's probably fun), so I do what I'm good at. Otherwise to say, \"stay in your lane or learn to love losing - but protect that ego at all costs and never admit fault ever!\" _(yes, I'm bitter and for exceptionally valid reasons)._\n\nSo, until the pre-IPO runway runs out, or the post-IPO environment bores people to tears and they jump ship with their ISOs and RSUs and START ALL OVER AGAIN! _(I have definitely never not done this exact thing several times never for sure for sure? sure.)_\n\n* * *\n\n## Successful Labs Require People and Hardware\n\nVaporware labs don't have hardware. Real labs do not revolve around PPTX slide decks, or Slack Canvas, or Sharepoint whatever, or ephemeral Confluence pages that change ownership by a click of a button. Real labs have real hardware and real engineers and architects involved at every critical stage to ensure that standards are being used, implemented, tested, and validated.\n\nIf you think I'm wrong, go ahead, do it all by slide deck, by half-baked simulators that don't even use SST and parallelization correctly, with lots of Big Discussions for the Big People you want to impress to get ahead instead of making quality hardware.\n\nSee how far the Peter Principle gets you - chances are, if you're surrounded by \"Yes Men\" and \"C-Suite but never Technical\" boardrooms, you will get far enough to be proven the fraud that you are.\n\n> The Peter principle is a concept in management .. which observes that people in a hierarchy tend to rise to \"a level of respective incompetence\": employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.\n\nHaving risen rapidly, since you're up high now - you can destroy entire companies by making extremely stupid choices _\"to save money\"_ by not spec'ing hot-swap trays for NVMe drives on 1U systems which were never designed for the wattage you're running - and were not spec'd for the BTU and LFM required to keep the systems from burning out 100G optics..\n\nJust when you think it's all good, um why are there only seven drives in this chassis that can run with eight?\n\n * Oh it's because the 8th drive would draw too much power and overload the non-redundant PSUs. What happens when a system which should have N+1 has only N?\n * The PSUs could be in redundant mode but Billy didn't want to authorize the enablement of active/active mode, and instead has demanded that his lacky make the systems run the PSUs in _\"combined mode\"_ to try to get more power to that 8th drive... but no... not possible.\n * So all of the fancy new systems can't have eight drives like the original spec REQUIRED in order to fulfill global fleet machine generation scaling models. As in, you lose 1/8th of your drives on every new system for so many thousands of systems. 1/8th, for those not inclined to fractional conversions, is\n * So now all of those systems will be missing ~2-16-64TB of NVMe drives (multiply by number of nodes per rack, per pod, per cage, per site, per region, per global fleet). That's a lot of expected high-speed storage that is now not being deployed.\n * Hopefully no contracts were made to customers based on _\"Total NVMe Raw Storage per <unit/region/etc>\"_ variables, which now renders the deliverable impossible.\n * What can be done? Those contracts have to either be revised and compensated _(yeah good f'in luck),_ or a massive workaround must be implemented to compensate - at additional cost to the company.\n\n\n\n### Why does that matter to building labs?\n\nBecause any lab worth their salt would have raised immediate red flags and halted the entire ill-devised scope. Then they would use standardized and periodically revised for betterment:\n\n * provide technical backing evidence\n * scaling costs, IOP/s cost, power assessments, baseline to STDDEV ratio, then appropriation wear-cycle equations\n * All that industry jargon that managerializes into the ether for all to repeat.\n * _\"keep me honest\"_ you may say, so then.. are you often lying? Ill-informed, unprepared, completely clueless?\n * Maybe not, hopefully not, but analytical people will see you that way and eventually the Peter Principle catches up and BAM! You're hit by a <REDACTED> and your entire team is fired, and so you move on with your RSUs, talk yourself up, and try to ruin those who smote you from afar.\n\n\n\nCongrats, Billy, you'll never change - **and that is why we test in labs** - **because things change**.\n\n* * *\n\n## The Contract Statement of Work\n\nAlways get the scope in writing with sign-off. Scope changes mid-stream, not awesome.\n\n\n Statement of Work\n\n Consultant Services Engagement\n\n This Statement of Work (\"SOW\") is entered into between [Client Legal\n Name] (\"Client\") and [Consultant Legal Name] (\"Consultant\"), effective\n as of [Effective Date], and is governed by the [Master Consulting\n Agreement / Professional Services Agreement] dated [Agreement Date] (the\n \"Agreement\"). In the event of any conflict between this SOW and the\n Agreement, the Agreement will control.\n\n -----------------------------------------------------------------------\n Client [Client Legal Name]\n ----------------------------------- -----------------------------------\n Consultant [Consultant Legal Name]\n\n SOW Effective Date [Effective Date]\n\n Initial Term Six (6) months from the SOW\n Effective Date\n\n Target Level of Effort Forty (40) hours per week,\n approximately 1,040 hours over the\n initial term\n\n Primary Stakeholders VP, OCTO, Eng-Org, App-Team, and\n Sales-Org\n\n Commercial Terms Billing rate, invoicing, payment\n terms, and approved expenses as set\n forth in the Agreement and/or\n applicable order form\n -----------------------------------------------------------------------\n\n 1. Purpose and Objectives\n\n The Consultant will provide technical leadership, architecture,\n implementation, documentation, mentoring, and subject matter expertise\n to establish standardized HPC benchmarking, workload automation, and\n telemetry/analytics capabilities, while also improving cross-functional\n execution and the overall developer experience for CORP products and\n associated collateral.\n\n - Define, document, and deploy a modular \"Industry Expectations\n Benchmark Suite\" leveraging HPC, SPEC, Top500, AI/ML GPU leaderboards,\n and SNIA methodologies to support rigorous adherence and consistent\n results using standard performance testing tools.\n\n - Define and deploy a standardized HPC \"Workload Automation Pipeline\"\n utilizing SLURM, Jenkins, Celery, RabbitMQ (\"RMQ\"), ClickHouse,\n Ansible, and related tooling.\n\n - Define and deploy a standardized HPC \"Telemetry + Analytics\n Aggregation + Data Visualization Processing Pipeline\" utilizing\n Check_MK, SNMP, Elasticsearch + Kibana, Prometheus, InfluxDB, Grafana,\n and related tooling.\n\n - Assess current organizational working patterns across Eng-Org,\n App-Team, and Sales-Org and provide a quarterly review to the VP and\n OCTO.\n\n - Assess the current CORP collateral surface area, including SDKs,\n documentation, libraries, proofs of concept, and reference designs, as\n an ongoing initiative to improve the developer experience.\n\n - Author white papers, research documents, and related technical\n materials, with a feedback loop to validate and improve internal and\n external documentation.\n\n - Provide mentoring for junior through senior application and systems\n engineers, plus reasonable availability as an SME across storage,\n systems, database, and AI/ML domains.\n\n - Provide ad hoc consultation to support Sales-Org customer scoping,\n implementation, and integration needs.\n\n 2. Term and Working Model\n\n The engagement will begin on the SOW Effective Date and continue for an\n initial six (6) month term. The Consultant will generally allocate forty\n (40) hours per week to the engagement, subject to mutually agreed\n priorities, holiday schedules, and reasonable scheduling adjustments.\n\n The parties acknowledge that some workstreams in this SOW are ongoing or\n iterative in nature. Accordingly, the Client and Consultant will jointly\n prioritize and sequence the work so that the highest-value deliverables\n are completed within the available time and level of effort.\n\n Services may be provided remotely or through other mutually agreed\n working arrangements. Any onsite work, travel, or work materially\n outside the target level of effort will require prior written approval.\n\n 3. Scope of Services\n\n 3.1 Modular \"Industry Expectations Benchmark Suite\"\n\n The Consultant will define, document, and deploy a modular benchmark\n suite intended to measure representative performance, reproducibility,\n and adherence to industry-standard methodologies across compute,\n storage, and AI/ML use cases.\n\n Illustrative deliverables:\n\n - Benchmark taxonomy and workload catalog covering applicable HPC, SPEC,\n Top500-style, AI/ML GPU leaderboard-relevant, and SNIA-aligned\n methodologies.\n\n - Benchmark execution standards, parameterization guidance, and runbooks\n designed to improve repeatability and result consistency.\n\n - Result templates and reporting conventions for capturing baseline\n measurements and comparative findings.\n\n - Initial implementation and/or deployment in designated Client\n environment(s), subject to infrastructure readiness and access.\n\n - Validation summary describing observed results, known limitations, and\n next-step recommendations.\n\n 3.2 Standardized HPC \"Workload Automation Pipeline\"\n\n The Consultant will define and deploy a standardized HPC workload\n automation architecture using SLURM, Jenkins, Celery, RMQ/RabbitMQ,\n ClickHouse, Ansible, and related tools as appropriate to the Client\n environment.\n\n Illustrative deliverables:\n\n - Reference architecture and component interaction design for job\n submission, orchestration, execution, retry handling, and result\n capture.\n\n - Deployment-ready configurations, automation artifacts, or initial\n implementation in designated environment(s).\n\n - Operational workflows for benchmark or workload scheduling, execution,\n data collection, and reporting.\n\n - Runbooks and handoff documentation sufficient to support repeatable\n operation and internal adoption.\n\n 3.3 Standardized HPC \"Telemetry + Analytics Aggregation + Data\n Visualization Processing Pipeline\"\n\n The Consultant will define and deploy a standardized observability and\n analytics processing pipeline spanning infrastructure telemetry,\n log/event aggregation, time-series monitoring, analytics storage, and\n visualization.\n\n Illustrative deliverables:\n\n - Reference architecture covering Check_MK, SNMP, Elasticsearch +\n Kibana, Prometheus, InfluxDB, Grafana, and related integrations as\n applicable.\n\n - Recommended data model, retention approach, dashboard structure, and\n operating practices for standardized reporting and analysis.\n\n - Initial dashboards, visualizations, and/or deployment artifacts for\n designated environment(s).\n\n - Documentation for data sources, ingestion paths, dashboards, and\n operational ownership considerations.\n\n 3.4 Cross-Functional Collaboration Assessment\n\n The Consultant will assess current organizational working patterns\n across Eng-Org, App-Team, and Sales-Org in order to identify friction\n points, handoff gaps, role ambiguity, communication issues, and\n opportunities to improve execution.\n\n Illustrative deliverables:\n\n - Current-state assessment and concise findings memo.\n\n - Recommendations for cadence, decision ownership, communication flow,\n and escalation patterns.\n\n - Quarterly review readout(s) presented to the VP and OCTO during the\n engagement term.\n\n 3.5 CORP Collateral Surface Area and Developer Experience\n\n The Consultant will assess the current ‘surface area’ of CORP collateral,\n including SDKs, documentation, libraries, proofs of concept, and\n reference designs, as an iterative stretch initiative with the overall\n goal of improving the developer experience.\n\n Illustrative deliverables:\n\n - Inventory and gap analysis of current CORP technical collateral.\n\n - Prioritized recommendations and/or backlog themes to improve\n consistency, usability, completeness, and customer/integrator\n experience.\n\n - Feedback loop for validating and improving internal and external\n documentation and supporting artifacts.\n\n - Periodic progress updates and next-step roadmap recommendations within\n the engagement window.\n\n 3.6 White Papers, Research Documents, and Documentation Improvement\n\n The Consultant will author white papers, research documents, technical\n notes, and similar written materials as prioritized by the Client, and\n will help create a feedback loop to validate and improve internal and\n external documentation quality.\n\n Illustrative deliverables:\n\n - White papers, technical briefs, research documents, architecture\n notes, or similar written outputs as mutually prioritized.\n\n - Document review comments, revision recommendations, and technical\n fact-checking input.\n\n - Structured feedback loop to capture lessons learned and improve future\n documentation quality.\n\n 3.7 Mentoring and Knowledge Transfer\n\n The Consultant will provide mentoring support for junior through senior\n application and systems engineers, with an emphasis on practical\n technical development and hands-on knowledge transfer.\n\n Illustrative deliverables:\n\n - Office hours, design reviews, pair-working sessions, workshops, or\n other mentoring interactions as appropriate.\n\n - Guidance on benchmarking, automation, observability, storage, systems,\n database, and AI/ML topics relevant to the engagement.\n\n - Knowledge transfer materials or summary notes where appropriate to\n support broader team adoption.\n\n 3.8 SME Availability and Sales-Org Consultation\n\n The Consultant will maintain reasonable availability to organizational\n staff in applicable subject-matter areas of focus and will provide ad\n hoc consultation for Sales-Org customer scoping, implementation, and\n integration needs, as capacity permits within the contracted level of\n effort.\n\n Illustrative deliverables:\n\n - Advisory support for storage, systems, database, and AI/ML questions\n arising during the engagement.\n\n - Participation in internal and customer-facing scoping discussions,\n solution reviews, or implementation planning sessions, when requested\n by the Client.\n\n - Technical input intended to improve solution fit, implementation\n readiness, and integration clarity; provided as advisory support\n rather than as a standalone managed service or customer support\n commitment.\n\n 4. Indicative Delivery Plan\n\n The following plan is illustrative and may be reprioritized by mutual\n agreement based on Client needs, environment readiness, and emerging\n business priorities.\n\n -----------------------------------------------------------------------\n Period Primary Focus Illustrative Outputs\n ----------------------- ----------------------- -----------------------\n Month 1 Discovery and Baseline Stakeholder alignment,\n current-state review,\n environment access,\n priority mapping,\n benchmark/workload\n definition, and\n architecture framing.\n\n Months 2-3 Design and Initial Benchmark suite v1,\n Deployment automation pipeline\n design and initial\n deployment artifacts,\n telemetry pipeline\n design, and initial\n dashboards or data\n paths.\n\n Months 4-5 Validation and Validation runs,\n Expansion documentation\n expansion,\n collaboration\n assessment findings,\n collateral review,\n white papers/research\n outputs, and mentoring\n support.\n\n Month 6 Handoff and Executive Runbooks,\n Review recommendations,\n backlog or roadmap\n items, knowledge\n transfer, and\n quarterly/executive\n review presentation(s)\n to the VP and OCTO.\n -----------------------------------------------------------------------\n\n 5. Governance and Reporting\n\n - The Client will designate a primary engagement owner or point of\n contact responsible for prioritization, coordination, and consolidated\n feedback.\n\n - The parties will maintain a regular working cadence, which may include\n weekly or biweekly syncs, technical reviews, design reviews, and\n status reporting as appropriate.\n\n - The Consultant will provide concise updates on progress, risks,\n blockers, and recommended next actions.\n\n - A quarterly review readout will be provided to the VP and OCTO during\n the engagement term.\n\n 6. Assumptions, Dependencies, and Constraints\n\n - The Client will provide timely access to relevant systems,\n environments, credentials, documentation, repositories, data, and\n subject-matter stakeholders reasonably required for the services.\n\n - The Client is responsible for providing or approving the hardware,\n software, licenses, network access, security approvals, and\n change-management windows required to implement or deploy\n deliverables.\n\n - Any production deployment, change control, or release activity remains\n subject to Client approval processes and operational safeguards.\n\n - Where this SOW references industry-standard benchmarks or\n leaderboard-adjacent methodologies, any formal external publication or\n third-party submission will remain subject to separate Client approval\n and third-party rules.\n\n - The Consultant’s work product may include recommendations, reference\n architectures, configurations, code or scripts, runbooks, dashboards,\n presentations, reports, and live working sessions; not every\n workstream will result in a standalone written deliverable.\n\n - Because certain activities are advisory and ongoing in nature, the\n parties will periodically rebalance priorities to remain within the\n contracted time and level of effort.\n\n - Travel, onsite presence, after-hours support, and work materially\n beyond the initial term or weekly level of effort are excluded unless\n separately approved in writing.\n\n 7. Out of Scope\n\n - Managed services, long-term production operations ownership, or 24x7\n operational support.\n\n - Hardware procurement, datacenter installation, or unrelated\n infrastructure build-out not directly tied to the services described\n in this SOW.\n\n - Binding commitments to end customers, contractual promises to third\n parties, or customer project delivery obligations not expressly\n authorized by the Client.\n\n - Ongoing maintenance, enhancement, or support of deliverables after the\n end of the engagement term, except as separately agreed in writing.\n\n 8. Acceptance and Change Control\n\n Commercial terms, invoicing, payment terms, intellectual property,\n confidentiality, warranties, limitation of liability, and any formal\n acceptance procedures will be governed by the Agreement.\n\n Where the Agreement does not specify a separate acceptance procedure,\n the Client will review deliverables promptly and provide consolidated\n feedback within five (5) business days after delivery. Deliverables will\n be deemed accepted when they materially conform to the scope described\n in this SOW, subject to any documented punch-list items or agreed\n revisions.\n\n Any material change to scope, deliverables, timeline, or level of effort\n will require written approval by both parties.\n\n 9. Signatures\n\n The parties acknowledge and agree to this SOW as of the dates written\n below.\n\n By: ________________________________\n Name: ______________________________\n\n\n [Consultant Legal Name]\n Title: _______________________________\n Date: _______________________________\n By: ________________________________\n\n [Client Legal Name]\n Name: ______________________________\n Title: _______________________________\n Date: _______________________________\n",
"title": "Let's Build a Lab - Part 2",
"updatedAt": "2026-06-11T03:26:37.419Z"
}