Proof of Skill Logo

Proof of Skill

Subject Matter Expert - Web Scraping & Crawling (Web Data Platform)

Posted 2 Months Ago
Be an Early Applicant
In-Office
Hyderabad, Telangana, IND
Expert/Leader
In-Office
Hyderabad, Telangana, IND
Expert/Leader
Own and productionize a web data acquisition platform combining resilient crawling with AWS Bedrock-based verification. Responsibilities include modernizing HTTP and browser crawling, implementing proxy and egress management, reverse-engineering APIs, respecting robots.txt and legal requirements, containerizing and scheduling pipelines, and building testing, observability, alerting, and canary-crawl systems. The role requires deep expertise across legacy and modern scraping stacks, anti-bot reliability, data extraction, distributed crawling, compliance, and large-scale platform ownership.
The summary above was generated by AI
About Chryselys

Chryselys is a Great Place to Work Certified Pharma Analytics & Business consulting company that delivers data-driven insights leveraging AI-powered, cloud-native platforms to achieve high-impact transformations. We specialize in digital technologies and advanced data science techniques that provide strategic and operational insights.

Role Summary

Own and extend a production-bound POC that combines resilient web-crawling engineering with an AWS Bedrock-based LLM verification agent, in a healthcare-data-compliance-sensitive context.

Responsibilities
  • Replace scraped SERPs with a paid search API behind the existing search_domain() interface.
  • Move fetch.py to async httpx with per-domain concurrency limits and politeness budgets.
  • Introduce proxy rotation and egress management; retire the single-IP failure mode.
  • Make failure loud: typed outcomes, structured logs, metrics, drift and volume alerts.
  • Reverse-engineer payer XHR/JSON endpoints to replace Playwright recipes wherever possible.
  • Honor robots.txt Disallow and Crawl-delay; build a per-domain ToS and licensing register.
  • Containerize and schedule the pipeline; add CI running the offline tests on every change.
  • Add HAR replay and golden-file parser tests on real payer HTML, plus daily canary crawls.
Skills

Web & protocol fundamentals — HTTP/1.1, HTTP/2, HTTP/3, TLS, JA3/JA4 fingerprinting, cookies/sessions, OAuth2/JWT/SSO, DOM, encodings [Must]

Legacy stack (real mileage) — urllib, requests, mechanize, cURL/wget, BeautifulSoup, lxml, html5lib, XPath/XSLT, Scrapy (middlewares, pipelines, CrawlSpider), Selenium 3 patterns, Splash/PhantomJS, RSS/Atom/SOAP/XML feeds, ASP.NET __VIEWSTATE postbacks, frame & table-layout scraping, sitemap.xml, WARC/Common Crawl [Must]

Modern stack — Playwright (contexts, tracing, network interception), Puppeteer, Selenium 4/CDP, httpx/aiohttp + asyncio, curl_cffi TLS impersonation, selectolax, scrapy-playwright, Crawlee/Apify, JSON-LD/microdata harvesting [Must]

Reverse engineering — Private/undocumented JSON APIs, XHR & fetch interception, GraphQL, mobile API capture (mitmproxy/Charles), bundled-JS deobfuscation, WASM challenge analysis [Must]

Methodology breadth — API-first vs browser rendering, static vs JS-rendered SPA (__NEXT_DATA__, Nuxt payloads), BFS/DFS frontier management, URL canonicalization & dedup (bloom filters/hashing), full-refresh vs incremental/CDC crawling, distributed queue-based crawling (Redis/Kafka/SQS), authenticated sessions, pagination & infinite scroll, per-domain politeness [Must]

Non-HTML extraction — PDF (pdfplumber, PyMuPDF — in use today), OCR (Tesseract/cloud), Office formats, LLM/vision-assisted extraction and its cost & failure modes [Must]

Anti-bot & reliability — Cloudflare, Akamai, DataDome, PerimeterX, Imperva, Kasada; browser/canvas/WebGL fingerprinting; residential/datacenter/mobile proxy rotation; CAPTCHA landscape; honeypot detection; backoff with jitter, circuit breakers, idempotent retries [Must]

Data engineering — Python expert (async, typing, profiling), SQL, pydantic validation (in use); pandera, entity resolution, Postgres/Mongo/S3/Parquet, Airflow/Prefect/Dagster, Docker/K8s, CI/CD to introduce; Node/TypeScript useful [Must]

Testing & observability — vcrpy/responses, HAR replay, golden-file parser tests, canary crawls, selector-drift alerting, volume-anomaly detection, structured logging & metrics [Must]

Build vs. buy — Evaluating Zyte, Bright Data, Oxylabs, Apify, ScrapingBee, Firecrawl on cost, coverage, and risk [Preferred]

Legal & ethical — robots.txt and rate-limit adherence, ToS/CFAA awareness, GDPR & PII minimization, licensed-API-first sourcing; designs defensible, low-risk acquisition strategies in partnership with Legal [Must]

Experience — Required
  • 8+ years in data acquisition; 5+ years owning a scraping platform end to end.
  • Proven scale: 1,000+ distinct domains or 10M+ pages/month sustained in production.
  • Rescued a brittle legacy scraper, with before/after reliability and cost numbers.
  • Mentored engineers; set crawl standards, review practice, and on-call runbooks.
  • Degree optional — equivalent practical experience is fully accepted.
Nice-to-Have
  • US payer policy, formulary, or prior-authorization document domain knowledge.
  • Open-source contributions to Scrapy, Playwright, Crawlee, or a parsing library.
  • LLM-assisted extraction at controlled cost — we run AWS Bedrock in verifier/.
  • Compliance or legal-review exposure on data acquisition programmes.
Equal Employment Opportunity

Chryselys is proud to be an Equal Employment Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.

Similar Jobs

An Hour Ago
In-Office
Hyderabad, Telangana, IND
Entry level
Entry level
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Develop and maintain full-stack applications using Angular, Java, SQL, REST APIs, microservices, Kafka, CI/CD, Docker, and Kubernetes. Build secure, scalable event-driven services, automate deployments, resolve production issues, and support features through production. Collaborate across teams, follow security and code-quality practices, and continuously adopt new technologies, including AI-related tools.
Top Skills: AngularApache KafkaCi/CdDevOpsDockerGitJavaKubernetesMicroservicesRestful ServicesSQL
4 Hours Ago
Remote or Hybrid
India
Mid level
Mid level
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Develop and execute embedded software verification plans and test cases for core platform components. Perform requirement- and scenario-based testing, change impact analysis, issue resolution, and technical audits. Collaborate with stakeholders, guide junior engineers, support continuous improvement, and ensure quality, cost, and schedule objectives. The role requires C/C++, RTOS, emulator and debugger, and verification-tool experience in an aerospace environment.
Top Skills: As9100BitbucketCC++DeosDoors NgIcdJIRALauterbachLdraLinuxRtosVectorcast/Cover
4 Hours Ago
Remote or Hybrid
India
Mid level
Mid level
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Develops and executes verification plans and test cases for embedded platform software. Performs requirements- and scenario-based testing, change impact analysis, technical issue resolution, audit support, and continuous improvement. The role collaborates with stakeholders, guides junior engineers, and uses embedded C/C++, RTOS platforms, emulators, debuggers, and verification tools.
Top Skills: As9100BitbucketCC++DeosDoors NgIcdJIRALauterbachLdraLinuxRtosVectorcast/Cover

What you need to know about the Hyderabad Tech Scene

Because of its proximity to leading research institutions and a government committed to the city's growth, Hyderabad's tech scene is booming. With plans to establish India's first "AI city," the city is on track to become one of the world's most anticipated tech hubs, with companies like TransUnion, Schrödinger and Freshworks, among others, already calling the city home.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account