Principal Software Engineer - Observability & Telemetry Data

The job

LIVE Employer public job feed (private-sector employer) Source: the employer's public careers feed · Data: listed 2026-09-01

Job description

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day.

Observability Engineering owns the breadth and depth of observability and telemetry at Smartsheet. We build and operate the shared platform every engineering team depends on to understand how their software behaves in production (metrics, logs, distributed traces, RUM, synthetics, and AI/agentic telemetry) across a global, multi-region, commercial and government footprint. We are consolidating ownership of all of it into a single correlated platform, and treating telemetry as what it actually is: one of the largest and most valuable datasets at the company. It's a broad mandate with real scope to shape, and we hold this internal platform to the same rigor, reliability, and product standards as customer-facing software. 

As a Principal Software Engineer (Observability & Telemetry Data), you will set the technical direction for how Smartsheet collects, models, stores, and derives value from telemetry data. This is an architect role: you will define the telemetry data platform and the standards that govern it, carry that architecture across engineering org boundaries, partner with our Data and AI Platform teams on Databricks and MLflow integration, and act as the company's definitive technical voice on observability. You will influence roadmaps you do not own, resolve the hard design trade-offs between fidelity and cost, and raise the technical bar for every engineer instrumenting a service. 

This full-time position reports to the Team Lead, Observability Engineering and can be located in our Bellevue, WA office, or you may work remotely from anywhere in the US where Smartsheet is a registered employer. 

You Will 

• Architect the Telemetry Data Platform: Own the end-to-end design for how telemetry is landed, modeled, and queried, including open table formats, partitioning and schema evolution strategy, separation of storage and compute, tiered retention, and the query interfaces engineers actually use. Make telemetry a durable, portable, Smartsheet-owned dataset rather than a vendor-locked byproduct. 
• Own the Telemetry Data Model: Define the semantic conventions, shared business identifiers (user, org, plan, tenant), and schema standards that let any signal be correlated with any other, and drive their adoption across every service team at Smartsheet. 
• Set OpenTelemetry Direction: Lead the migration to OTel-based instrumentation, defining collector architecture, context propagation, and sampling strategy (including tail-based sampling) so that engineers can move from a log line to a trace to a metric without losing the thread.
• Integrate AI and Agentic Telemetry with Databricks: Own the architecture connecting our observability platform to Databricks and MLflow, so that agentic and model telemetry (prompt, completion, tool and MCP calls, evaluation results) is captured un-sampled, stays useful under the input/output redaction our governance requires, and reconciles cleanly with the traces and metrics in our primary observability stack. 
• Instrument the Data Platform Itself: Bring first-class observability to our data estate, including Databricks jobs, pipelines, and warehouses, with meaningful signals for freshness, data quality, lineage, and cost, so data reliability is measured with the same discipline as service reliability. 
• Build the Analytics Layer on Telemetry: Turn telemetry into decision-grade analytics, covering reliability and incident metrics, telemetry cost and chargeback models, and adoption and coverage reporting that leadership can act on. 
• Engineer Collection and Routing at Scale: Architect the high-volume collection and routing tier (FluentBit, Kinesis, and OTel collectors) that moves telemetry from every service to its destination across US, EU, AU, and GovCloud regions, and own the migration of these pipelines as we consolidate onto a unified backend. 
• Own Telemetry Economics: Set the cost architecture for observability data, including ingest governance, cardinality control, and storage tiering, so that teams get the fidelity they need to debug without the spend pressure that causes them to under-instrument. 
• Carry Architecture Across Org Boundaries: Partner with the Data Platform, AI Platform, and infrastructure organizations to align telemetry architecture with theirs, influence roadmaps you do not own, and represent observability in company-level platform and vendor decisions. 
• Raise the Technical Bar: Lead design and code reviews, author the architecture decisions and standards others build against, and mentor senior and mid-level engineers on instrumentation, telemetry data modeling, and cost-aware design. 
• Participate in a production support and on-call rotation, taking ownership of the most complex issue resolution and driving root-cause analysis that improves system resiliency. 

You Have 

• 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure, including time at Principal or Staff level. 
• Deep Observability Expertise: Hands-on production ownership of metrics, logs, and distributed tracing at scale, including at least one major backend (Datadog, or comparable) and a working understanding of cardinality and cost mechanics. 
• Telemetry Data Engineering: Demonstrated depth in large-scale data architecture, including open table formats (Delta Lake, Apache Iceberg), Spark or comparable distributed processing, streaming ingestion, partitioning and schema evolution, and query performance and cost tuning over very large datasets. 
• Databricks Depth: Practical experience with the Databricks platform (jobs, clusters, Unity Catalog, Delta) and with MLflow for model and agent telemetry. 
• OpenTelemetry Depth: Practical experience with OTel collectors, semantic conventions, context propagation, and sampling strategy, including tail-based sampling. 
• Pipeline Engineering: Experience with high-volume log and telemetry pipelines, including FluentBit or Fluentd, streaming transport such as Kinesis or Kafka, and search backend index and mapping design.
• Advanced AWS & Kubernetes Expertise: EKS, ECS Fargate, EC2, Lambda, and CloudWatch in production. 
• 10+ years of programming experience with modern languages such as Go, Java, Python, or Scala, and strong SQL. 
• Infrastructure as Code: Terraform, and GitOps workflows such as Flux or ArgoCD. 
• Architectural Influence at Scale: A track record of setting technical direction that multiple teams and organizations adopted, including the written artifacts (architecture decisions, standards, RFCs) that made it durable after you moved on. 
• Strong incident response instincts, with experience improving mean-time-to-resolution through better instrumentation and better data rather than more heroics. 
• A degree in Computer Science, Engineering, or a related field, or equivalent practical industry experience.
• Legally eligible to work in the U.S. on an ongoing basis 

Nice to Have 

• Experience instrumenting LLM or agentic systems, including OTel GenAI semantic conventions and tracing agent tool-call workflows. 
• Data reliability engineering practice: freshness, quality, and lineage SLOs for production data pipelines
• Telemetry cost engineering or FinOps at scale. 
• Experience in regulated environments (FedRAMP, GovCloud) and designing telemetry that remains useful under redaction. 
• Prometheus, Grafana, and Alertmanager. 
• Snowflake, or experience operating across more than one lakehouse or warehouse platform. 
• Service catalog or internal developer platform work (Backstage or similar). 

Current US Perks & Benefits:

• Employer subsidized medical/vision and dental coverage for full-time employees
• 401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
• Monthly stipend to support your work and productivity
• Flexible Time Away Program, plus Sick Time Off
• US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
• US employees receive 12 paid holidays per year
• Up to 24 weeks of Parental Leave
• Personal paid Volunteer Day to support our community
• Opportunities for professional growth and development including access to Udemy online courses
• Company Funded Perks, including a counseling membership, local retail discounts, and your own personal Smartsheet account
• Teleworking options from any registered location in the U.S. (role specific)

Smartsheet provides a competitive base salary range for roles that may be hired in different geographic areas we are licensed to operate our business from. Actual compensation is determined by several factors including, but not limited to, level of professional, educational experience, skills, and specific candidate location. In addition, this role will be eligible for a market competitive incentive opportunity.

Get to Know Us:

At Smartsheet, your ideas are heard, your potential is supported, and your contributions have real impact. You’ll have the freedom to explore, push boundaries, and grow beyond your role. We welcome diverse perspectives and nontraditional paths—because we know that impact comes from individuals who care deeply and challenge thoughtfully. When you’re doing work that stretches you, excites you, and connects you to something bigger, that’s magic at work. Let’s build what’s next, together.

Equal Opportunity Employer:

Smartsheet is an Equal Opportunity (EEO) employer committed to fostering an inclusive environment with the best employees. It is our policy to provide equal employment opportunities to all qualified applicants in accordance with applicable laws in the US, UK, Australia, Germany, Costa Rica, Japan, Bulgaria, India, and Singapore. All qualified applicants will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran or disabled status, or genetic information. 

If there are preparations we can make to help ensure you have a comfortable and positive interview experience, please let us know.

#LI-Remote

Explore around this job

Field: Technology Sub-niche: Software engineering All jobs at Smartsheet Jobs in Washington Jobs in Bellevue, WA Bellevue city guide Washington state guide King County economy

View this job and apply on the employer's page →

You apply on the employer's page; AtiveSite does not receive or process applications and is not affiliated with the employer. The description above is shown as the source published it.

About the role

This is a — role in the Technology field (Software engineering) at Smartsheet.

We could not match this job title to one occupation in the official wage statistics, so no pay estimate is shown. Pay is not in the source listing; ask the employer for the range and compare in the 240 job niches.

Does the pay work here? (estimate)

How to prepare and behave

This is editorial guidance, not information from the employer.

• Read the whole description and match your examples to its requirements; use the same words the posting uses where they are true for you.
• Research the employer before the interview: what it does, who its customers are and recent news; prepare two or three questions about the work.
• Arrive early (or log in early for a video call), dress one step above the daily dress code, and test your camera and audio beforehand.
• Answer with short stories: the situation, what you did and the result. Give numbers when you have them and never invent them.
• Be honest about gaps; say what you did to learn and how fast you can close them.
• Ask about the team, the first 90 days, how success is measured and the schedule; do not open with salary and time off.
• When pay comes up, know the market range for the occupation and your lowest acceptable offer before you answer; ask for the full package in writing.
• Send a short thank-you message within a day, mentioning one thing you discussed.
• Have code, data work or projects you can show, and be ready to think aloud: many technical interviews test how you reason, not only the answer.

Know your rights as a candidate and an employee:

• U.S. Equal Employment Opportunity Commission: prohibited employment practices
• U.S. Department of Labor: Wage and Hour Division
• CareerOneStop (U.S. Department of Labor): job search help, interview tips and resumes
• USA.gov: avoiding job scams

The city

The job is in Bellevue, WA. Below is what the official statistics say about living and working there.

How Bellevue compares. Among the 80 largest places in Washington, Bellevue is #5 by population, #5 for college degrees (#1 = most), #75 for rent (#1 = cheapest), #7 for household income (#1 = highest). Its county, King County, is #1 of 39 counties for average pay ($132,279).

The place in numbers

Bellevue has 151,847 residents (ACS 2020–2024), about 1.9% of the population of Washington. It covers 33 square miles of land, which works out to roughly 4,537 people per square mile. The median age is 38.5, about the same as the state median (38.5 against 38.3).

Housing

The median gross rent is $2,572 a month (rent plus utilities), 46% above the state median ($2,572 against $1,760). Over a year that is 18.6% of the median household income. US housing guidance treats housing as affordable when it takes 30% of income or less, so the typical household is under that line. The median home value is $1,340,300, which is 8.1 times the median household income ($165,576); a ratio above 5 usually means buying is out of reach without savings or a second income. 52.0% of households own their home, 18% below the state average (52.0% against 63.8%).

Work and pay

The Seattle-Tacoma-Bellevue, WA metropolitan area has 2,086,210 jobs in the BLS wage survey, and the median pay across all occupations is $72,610 a year, 42% above the national median ($72,610 against $50,980). The biggest occupation groups are office and administrative support (212,900 jobs, median $59,660); business and financial operations (201,160 jobs, median $101,950); computer and mathematical (194,970 jobs, median $156,000). The area is most specialised in computer and mathematical (2.8 times) and architecture and engineering (1.4 times), where the share of workers is well above the national share. The best-paying groups with at least 5,000 jobs are management ($168,110) and computer and mathematical ($156,000). In 2026M09 the unemployment rate of King County was 4.8%. 67.0% of residents aged 16 and over are in the labour force. The city lies in King County; its jobs, pay by industry and GDP are on the county page.

Getting around

Among workers, 45.6% drive alone, 5.1% use public transit, 4.5% walk, 35.5% work from home. 11.1% of workers commute 45 minutes or more, 34% below the state average (11.1% against 16.9%). Remote work is more common than in the state as a whole, which widens the set of employers a resident can work for without moving.

People and education

72.1% of adults aged 25 and over hold a bachelor's degree or higher, 82% above the state share (72.1% against 39.6%). 43.5% of residents were born outside the United States, 181% above the state share (43.5% against 15.4%). The poverty rate is 7.2%, 27% below the state rate (7.2% against 9.9%).

Climate

The warmest month is Aug, averaging 63°F (17.1°C), and the coolest is Dec at 34°F (1.0°C). About 1,737 mm (68 inches) of precipitation falls in a year; Nov is the wettest month (256 mm) and Jul the driest (27 mm). These values come from NASA satellite-based climatology on a grid of about 50 km, so they describe the wider area, not a weather station in the city.

Nearby cities to compare

Closest cities in this guide: Mercer Island (4 miles, similar rent); Sammamish (6 miles, rent 11% higher); Redmond (6 miles, rent 6% lower); Issaquah (6 miles, similar rent). Rent differences are against Bellevue; use the Compare tab to put any two cities side by side.

Full guide to Bellevue →Washington guideAll jobs in Bellevue

Official links for Bellevue

• Official website of Bellevue: bellevuewa.gov
• Census Bureau data profile for Bellevue
• Official state website: wa.gov
• BLS Economy at a Glance: Washington

These links go to official sources and open in a new window. The first one is the government's or company's own site as listed in Wikidata (property "official website"); check it before relying on it.

Where to live

CITY Official statistics Source: U.S. Census Bureau, American Community Survey 2020–2024 5-year (via Census Reporter) · Data: ACS 2020–2024

Places within 50 miles (straight line) of Bellevue, WA that have a guide, with what living there costs. No pay estimate is shown because the title did not match one occupation.

Career paths

We could not match this title to one occupation, so here are the other sub-niches in Technology, where your skills may carry over:

Cybersecurity (512) Data, analytics and AI (2461) Cloud, IT and infrastructure (743) Product management (570)

Distances

CITY Calculation (straight line) Source: U.S. Census Bureau Gazetteer (coordinates) · Data: 2024

How far Bellevue, WA is from the 20 largest US cities, in straight-line miles (not driving distance). Click a column to sort.

Nearby cities with open jobs (within 120 miles)

The employer

Smartsheet is the employer in this listing. See all its open jobs here →

We do not hold headcount or revenue for this employer. Check its careers site and, if it is listed on a stock exchange, its public filings (SEC EDGAR search).

What it is like to work at a company (culture, schedules, benefits) is not in the source data, so we do not describe it. Read the description and ask the questions under "How to prepare".

Other open jobs at Smartsheet

Similar jobs

Same sub-niche: Software engineering

More jobs in Bellevue, WA

All Software engineering jobs → · All Technology sub-niches →

FAQ

What is the Principal Software Engineer - Observability & Telemetry Data job at Smartsheet?

A private-sector role in technology (software engineering) listed 69 days ago. The full description is on this page and on the employer's page.

Where is this job?

Bellevue, WA.

How do I apply?

Use the "View this job and apply on the employer's page" button. AtiveSite does not receive applications.

How much does it pay?

The listing does not publish pay, and we could not match the title to one official occupation.

What is the cost of living there?

Median rent in Bellevue is $2,572 a month and median household income $165,576 (ACS 2020–2024).

How old is this listing?

It was posted on 2026-09-01 (69 days ago). Jobs older than 90 days are not shown.

Are there similar jobs?

Yes: browse software engineering jobs in Washington.

Sources and terms

• Job data: Smartsheet careers page public feed (listing dated 2026-09-01). The job disappears from this site when it disappears from the source.
• Wages and projections: BLS OEWS (May 2025) and Employment Projections 2025–35.
• City and county figures: Census Bureau ACS 2020–2024, BEA, BLS QCEW and LAUS, NASA POWER climatology; distances from Census Gazetteer coordinates.

AtiveSite is not affiliated with or endorsed by the employer or any government agency, and does not process applications. Verify every detail with the employer before acting.

Sources

• BLS OEWS (wages and employment by occupation, May 2025)
• BLS Employment Projections 2025–35
• BLS Local Area Unemployment Statistics
• BLS JOLTS
• BLS State and Metro Area Employment
• Census Bureau American Community Survey (via Census Reporter)
• BEA GDP by state and Regional Price Parities
• IRS 2026 tax adjustments
• U.S. Department of Labor, minimum wage by state
• Tax Foundation state income tax rates 2026
• USAJOBS
• USCIS
• U.S. Department of State, visas
• NASA POWER climatology
• GeoNames / Census Gazetteer (coordinates)