Data Scientist

The role

The Data Scientist will support applied machine learning, natural language processing, AI evaluation, data quality, and analytical product development for a federal permitting modernization initiative. The role involves working across structured and unstructured data to develop evaluation datasets and quantitative methods for AI-enabled and agentic solutions.

What you will do

• Develop analytical and machine-learning approaches supporting permitting and natural-resource use cases.
• Build and maintain labeled datasets, evaluation corpora, and benchmark datasets for AI-enabled services.
• Define quantitative evaluation methodologies for AI/NLP capabilities.
• Evaluate LLM and agentic workflows including output quality, retrieval performance, task completion, failure modes, and reliability.
• Analyze public-comment datasets and develop reusable features and evaluation datasets.
• Apply NLP and machine-learning techniques to structured and unstructured data.
• Evaluate embedding models, semantic retrieval strategies, document-ranking techniques, and AI-generated outputs.
• Develop statistical approaches for detecting anomalies, reconciliation issues, missing data, and inconsistent records.
• Partner with Content Pipeline Engineers to measure OCR, extraction, and document-retrieval performance.
• Partner with Data Engineers to ensure analytical datasets and evaluation pipelines are reproducible and production-ready.

Required qualifications

• Bachelor's degree in Data Science, Statistics, Computer Science, Mathematics, Engineering, or a related quantitative discipline.
• 5+ years of relevant data science, applied machine learning, or quantitative analytics experience.
• Strong Python and SQL skills.
• Experience with statistical analysis, machine learning, NLP, and unstructured text data.
• Experience designing labeled datasets, benchmark datasets, and train/test evaluation frameworks.
• Experience evaluating LLM, RAG, or agentic AI solutions, including retrieval quality, output accuracy, and failure modes.
• Strong understanding of model-evaluation techniques, including precision, recall, F1, ranking metrics, confusion matrices, and related measures.
• Experience with embeddings, semantic retrieval, or vector-based search.
• Experience developing reproducible analytical and evaluation workflows.
• Ability to distinguish data-quality issues from model-performance issues.
• Strong ability to communicate analytical results, limitations, and uncertainty.

Nice to have

• Experience with agent evaluation frameworks or LLM observability tools.
• Experience with agentic frameworks such as LangChain, LangGraph, or similar orchestration frameworks.
• Experience with OpenSearch, vector databases, or enterprise search platforms.
• Experience with document AI or OCR pipelines.
• Experience with geospatial analysis.
• Experience developing entity-resolution, record-linkage, or data-quality models.
• Experience working in the U.S. Federal Government space.
• Experience with permitting, regulatory, environmental, energy, or natural-resource datasets.

Benefits

• medical, dental and vision coverage
• 401k plan with employer contribution
• paid holidays
• vacation
• tuition reimbursement

About the company

blackcanyonconsulting is a high performing, professional service and technology focused organization.