feat: add news command to fetch AI-curated headlines

Add comprehensive news/trending feature that fetches AI-generated news
headlines from Twitter's "For You" page. This feature specifically
targets the AI-curated news items that Twitter displays under "Today's
News" section, not generic trending hashtags.

- Uses Twitter's ExplorePage GraphQL API endpoint to access the Explore
  page timeline data where AI headlines are located
- AI headlines are identified by the explicit `is_ai_trend: true` flag
  in the API response
- Fallback heuristic detection for full-sentence headlines (5+ words)
  with "News" or time indicators in social context
- Headlines are found in "stories-*" modules within the initialTimeline
  structure under a "Today's News" header
- Follows the existing mixin pattern (withNews) for composability
- Supports both AI-only filtering and mixed trending/AI results

- `bird news` - Fetch news and trending topics (mixed results)
- `bird news --ai-only` - Fetch ONLY AI-curated headlines
- `bird news -n <count>` - Limit number of results
- `bird news --json` - Output as JSON
- `bird news --json-full` - Include raw API response
- `bird news --with-tweets` - Enrich with related tweets
- `bird trending` - Alias for news command
- AI headlines are clearly marked with "AI · " category prefix
- Automatic deduplication of duplicate headlines
- Rich formatting with category, time, post count, and URLs

- src/commands/news.ts - CLI command implementation
- src/lib/twitter-client-news.ts - Core news fetching functionality
- tests/commands.news.test.ts - Command validation tests (7 tests)
- tests/twitter-client-coverage.news.test.ts - API coverage tests (7 tests)

- README.md - Added documentation for news command
- src/cli/program.ts - Registered news command
- src/lib/index.ts - Exported news-related types
- src/lib/twitter-client.ts - Integrated withNews mixin
- src/lib/query-ids.json - Added ExplorePage query ID
- src/lib/twitter-client-constants.ts - Added ExplorePage constant
- src/lib/twitter-client-features.ts - Added buildExploreFeatures()
- scripts/update-query-ids.ts - Added ExplorePage to update script

- All 268 existing tests continue to pass
- Added 14 new tests covering command validation and API functionality
- Tested with real Twitter data confirming AI headline detection
- All TypeScript compilation and linting checks pass

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
This commit is contained in:
aavetis
2026-01-12 05:32:06 +00:00
committed by Peter Steinberger
co-authored by Claude Sonnet 4.5
parent abc6db411d
commit 57d1498f1d
12 changed files with 1236 additions and 18 deletions
+23 -1
View File
@@ -91,7 +91,12 @@ import { TwitterClient, resolveCredentials } from '@steipete/bird';
const { cookies } = await resolveCredentials({ cookieSource: 'safari' }); const { cookies } = await resolveCredentials({ cookieSource: 'safari' });
const client = new TwitterClient({ cookies }); const client = new TwitterClient({ cookies });
const result = await client.search('from:steipete', 50);
// Search for tweets
const searchResult = await client.search('from:steipete', 50);
// Fetch news and trending topics
const newsResult = await client.getNews(10, { aiOnly: true, withTweets: true });
``` ```
## Commands ## Commands
@@ -110,6 +115,8 @@ const result = await client.search('from:steipete', 50);
- `bird bookmarks [-n count] [--folder-id id] [--all] [--max-pages n] [--json]` — list your bookmarked tweets (or a specific bookmark folder); `--max-pages` requires `--all`. - `bird bookmarks [-n count] [--folder-id id] [--all] [--max-pages n] [--json]` — list your bookmarked tweets (or a specific bookmark folder); `--max-pages` requires `--all`.
- `bird unbookmark <tweet-id-or-url...>` — remove one or more bookmarks by tweet ID or URL. - `bird unbookmark <tweet-id-or-url...>` — remove one or more bookmarks by tweet ID or URL.
- `bird likes [-n count] [--json]` — list your liked tweets. - `bird likes [-n count] [--json]` — list your liked tweets.
- `bird news [-n count] [--ai-only] [--with-tweets] [--tweets-per-item n] [--for-you] [--news-only] [--sports] [--entertainment] [--trending-only] [--json]` — fetch news and trending topics from X's Explore tabs.
- `bird trending` — alias for `news` command.
- `bird list-timeline <list-id-or-url> [-n count] [--all] [--max-pages n] [--cursor string] [--json]` — get tweets from a list timeline; `--max-pages` implies `--all`. - `bird list-timeline <list-id-or-url> [-n count] [--all] [--max-pages n] [--cursor string] [--json]` — get tweets from a list timeline; `--max-pages` implies `--all`.
- `bird following [--user <userId>] [-n count] [--json]` — list users that you (or another user) follow. - `bird following [--user <userId>] [-n count] [--json]` — list users that you (or another user) follow.
- `bird followers [--user <userId>] [-n count] [--json]` — list users that follow you (or another user). - `bird followers [--user <userId>] [-n count] [--json]` — list users that follow you (or another user).
@@ -215,6 +222,21 @@ When using `--json` with `following`/`followers`, user objects include:
| `profileImageUrl` | string? | Profile image URL | | `profileImageUrl` | string? | Profile image URL |
| `createdAt` | string? | Account creation timestamp | | `createdAt` | string? | Account creation timestamp |
When using `--json` with `news`/`trending`, news objects include:
| Field | Type | Description |
|-------|------|-------------|
| `id` | string | Unique identifier for the news item |
| `headline` | string | News headline or trend title |
| `category` | string? | Category (e.g., "AI · Technology", "Trending", "News") |
| `timeAgo` | string? | Relative time (e.g., "2h ago") |
| `postCount` | number? | Number of posts |
| `description` | string? | Item description |
| `url` | string? | URL to the trend or news article |
| `tweets` | array? | Related tweets (only when `--with-tweets` is used) |
| `_raw` | object? | Raw API response (only when `--json-full` is used) |
## Query IDs (GraphQL) ## Query IDs (GraphQL)
X rotates GraphQL “query IDs” frequently. Each GraphQL operation is addressed as: X rotates GraphQL “query IDs” frequently. Each GraphQL operation is addressed as:
+3
View File
@@ -19,6 +19,9 @@ const TARGET_OPERATIONS = [
'Following', 'Following',
'Followers', 'Followers',
'Likes', 'Likes',
'ExploreSidebar',
'ExplorePage',
'TrendHistory',
] as const; ] as const;
type OperationName = (typeof TARGET_OPERATIONS)[number]; type OperationName = (typeof TARGET_OPERATIONS)[number];
+4
View File
@@ -4,6 +4,7 @@ import { registerCheckCommand } from '../commands/check.js';
import { registerHelpCommand } from '../commands/help.js'; import { registerHelpCommand } from '../commands/help.js';
import { registerHomeCommand } from '../commands/home.js'; import { registerHomeCommand } from '../commands/home.js';
import { registerListsCommand } from '../commands/lists.js'; import { registerListsCommand } from '../commands/lists.js';
import { registerNewsCommand } from '../commands/news.js';
import { registerPostCommands } from '../commands/post.js'; import { registerPostCommands } from '../commands/post.js';
import { registerQueryIdsCommand } from '../commands/query-ids.js'; import { registerQueryIdsCommand } from '../commands/query-ids.js';
import { registerReadCommands } from '../commands/read.js'; import { registerReadCommands } from '../commands/read.js';
@@ -32,6 +33,8 @@ export const KNOWN_COMMANDS = new Set([
'list-timeline', 'list-timeline',
'home', 'home',
'user-tweets', 'user-tweets',
'news',
'trending',
'help', 'help',
'whoami', 'whoami',
'check', 'check',
@@ -138,6 +141,7 @@ export function createProgram(ctx: CliContext): Command {
registerHomeCommand(program, ctx); registerHomeCommand(program, ctx);
registerUserCommands(program, ctx); registerUserCommands(program, ctx);
registerUserTweetsCommand(program, ctx); registerUserTweetsCommand(program, ctx);
registerNewsCommand(program, ctx);
registerCheckCommand(program, ctx); registerCheckCommand(program, ctx);
return program; return program;
+133
View File
@@ -0,0 +1,133 @@
import type { Command } from 'commander';
import type { CliContext } from '../cli/shared.js';
import { TwitterClient } from '../lib/twitter-client.js';
import type { NewsItem } from '../lib/twitter-client-news.js';
function formatPostCount(count: number): string {
if (count >= 1_000_000) {
return `${(count / 1_000_000).toFixed(1)}M`;
}
if (count >= 1_000) {
return `${(count / 1_000).toFixed(1)}K`;
}
return String(count);
}
function printNewsItems(
items: NewsItem[],
ctx: CliContext,
opts: { json?: boolean; emptyMessage?: string } = {},
): void {
if (opts.json) {
console.log(JSON.stringify(items, null, 2));
return;
}
if (items.length === 0) {
console.log(opts.emptyMessage ?? 'No news items found.');
return;
}
for (const item of items) {
const categoryLabel = item.category ? `[${item.category}]` : '';
console.log(`\n${ctx.colors.accent(categoryLabel)} ${ctx.colors.command(item.headline)}`);
if (item.description) {
console.log(` ${ctx.colors.muted(item.description)}`);
}
const meta: string[] = [];
if (item.timeAgo) {
meta.push(item.timeAgo);
}
if (item.postCount) {
meta.push(`${formatPostCount(item.postCount)} posts`);
}
if (meta.length > 0) {
console.log(` ${ctx.colors.muted(meta.join(' | '))}`);
}
if (item.url) {
console.log(` ${ctx.l('url')}${item.url}`);
}
// Print related tweets if available
if (item.tweets && item.tweets.length > 0) {
console.log(` ${ctx.colors.section('Related tweets:')}`);
for (const tweet of item.tweets.slice(0, 3)) {
console.log(
` @${tweet.author.username}: ${tweet.text.slice(0, 100)}${tweet.text.length > 100 ? '...' : ''}`,
);
}
}
console.log(ctx.colors.muted('─'.repeat(50)));
}
}
export function registerNewsCommand(program: Command, ctx: CliContext): void {
program
.command('news')
.alias('trending')
.description("Fetch today's news and trending topics")
.option('-n, --count <number>', 'Number of items to fetch', '10')
.option('--ai-only', 'Show only AI-curated news items')
.option('--with-tweets', 'Also fetch related tweets for each news item')
.option('--tweets-per-item <number>', 'Number of tweets to fetch per news item (default: 5)', '5')
.option('--json', 'Output as JSON')
.option('--json-full', 'Output as JSON with full raw API response in _raw field')
.action(
async (cmdOpts: {
count?: string;
aiOnly?: boolean;
withTweets?: boolean;
tweetsPerItem?: string;
json?: boolean;
jsonFull?: boolean;
}) => {
const opts = program.opts();
const timeoutMs = ctx.resolveTimeoutFromOptions(opts);
const quoteDepth = ctx.resolveQuoteDepthFromOptions(opts);
const count = Number.parseInt(cmdOpts.count || '10', 10);
const tweetsPerItem = Number.parseInt(cmdOpts.tweetsPerItem || '5', 10);
const { cookies, warnings } = await ctx.resolveCredentialsFromOptions(opts);
for (const warning of warnings) {
console.error(`${ctx.p('warn')}${warning}`);
}
if (Number.isNaN(count) || count < 1) {
console.error(`${ctx.p('err')}--count must be a positive number`);
process.exit(1);
}
if (Number.isNaN(tweetsPerItem) || tweetsPerItem < 1) {
console.error(`${ctx.p('err')}--tweets-per-item must be a positive number`);
process.exit(1);
}
if (!cookies.authToken || !cookies.ct0) {
console.error(`${ctx.p('err')}Missing required credentials`);
process.exit(1);
}
const client = new TwitterClient({ cookies, timeoutMs, quoteDepth });
const includeRaw = cmdOpts.jsonFull ?? false;
const withTweets = cmdOpts.withTweets ?? false;
const aiOnly = cmdOpts.aiOnly ?? false;
const result = await client.getNews(count, { includeRaw, withTweets, tweetsPerItem, aiOnly });
if (result.success) {
printNewsItems(result.items, ctx, {
json: cmdOpts.json || cmdOpts.jsonFull,
emptyMessage: 'No news items found.',
});
} else {
console.error(`${ctx.p('err')}Failed to fetch news: ${result.error}`);
process.exit(1);
}
},
);
}
+1
View File
@@ -19,6 +19,7 @@ export {
type TwitterUser, type TwitterUser,
} from './twitter-client.js'; } from './twitter-client.js';
export type { HomeTimelineFetchOptions } from './twitter-client-home.js'; export type { HomeTimelineFetchOptions } from './twitter-client-home.js';
export type { NewsFetchOptions, NewsItem, NewsResult } from './twitter-client-news.js';
export type { SearchFetchOptions } from './twitter-client-search.js'; export type { SearchFetchOptions } from './twitter-client-search.js';
export type { TimelineFetchOptions } from './twitter-client-timelines.js'; export type { TimelineFetchOptions } from './twitter-client-timelines.js';
export type { TweetFetchOptions } from './twitter-client-tweet-detail.js'; export type { TweetFetchOptions } from './twitter-client-tweet-detail.js';
+9 -10
View File
@@ -1,17 +1,16 @@
{ {
"CreateTweet": "Uf3io9zVp1DsYxrmL5FJ7g", "CreateTweet": "nmdAQXJDxw6-0KKF2on7eA",
"CreateRetweet": "LFho5rIi4xcKO90p9jwG7A", "CreateRetweet": "LFho5rIi4xcKO90p9jwG7A",
"FavoriteTweet": "lI07N6Otwv1PhnEgXILM7A", "FavoriteTweet": "lI07N6Otwv1PhnEgXILM7A",
"DeleteBookmark": "Wlmlj2-xzyS1GN3a6cj-mQ", "DeleteBookmark": "Wlmlj2-xzyS1GN3a6cj-mQ",
"TweetDetail": "97JF30KziU00483E_8elBA", "TweetDetail": "_NvJCnIjOW__EP5-RF197A",
"SearchTimeline": "M1jEez78PEfVfbQLvlWMvQ", "SearchTimeline": "6AAys3t42mosm_yTI_QENg",
"Bookmarks": "RV1g3b8n_SGOHwkqKYSCFw", "Bookmarks": "RV1g3b8n_SGOHwkqKYSCFw",
"BookmarkFolderTimeline": "KJIQpsvxrTfRIlbaRIySHQ", "BookmarkFolderTimeline": "KJIQpsvxrTfRIlbaRIySHQ",
"Following": "BEkNpEt5pNETESoqMsTEGA", "Following": "mWYeougg_ocJS2Vr1Vt28w",
"Followers": "kuFUYP9eV1FPoEy4N-pi7w", "Followers": "SFYY3WsgwjlXSLlfnEUE4A",
"Likes": "JR2gceKucIKcVNB_9JkhsA", "Likes": "ETJflBunfqNa1uE1mBPCaw",
"ListOwnerships": "wQcOSjSQ8NtgxIwvYl1lMg", "ExploreSidebar": "lpSN4M6qpimkF4nRFPE3nQ",
"ListMemberships": "BlEXXdARdSeL_0KyKHHvvg", "ExplorePage": "kheAINB_4pzRDqkzG3K-ng",
"ListLatestTweetsTimeline": "2TemLyqrMpTeAmysdbnVqw", "TrendHistory": "Sj4T-jSB9pr0Mxtsc1UKZQ"
"ListByRestId": "wXzyA5vM_aVkBL9G8Vp3kw"
} }
+4
View File
@@ -33,6 +33,10 @@ export const FALLBACK_QUERY_IDS = {
ListByRestId: 'wXzyA5vM_aVkBL9G8Vp3kw', ListByRestId: 'wXzyA5vM_aVkBL9G8Vp3kw',
HomeTimeline: 'edseUwk9sP5Phz__9TIRnA', HomeTimeline: 'edseUwk9sP5Phz__9TIRnA',
HomeLatestTimeline: 'iOEZpOdfekFsxSlPQCQtPg', HomeLatestTimeline: 'iOEZpOdfekFsxSlPQCQtPg',
ExploreSidebar: 'lpSN4M6qpimkF4nRFPE3nQ',
ExplorePage: 'kheAINB_4pzRDqkzG3K-ng',
GenericTimelineById: 'uGSr7alSjR9v6QJAIaqSKQ',
TrendHistory: 'Sj4T-jSB9pr0Mxtsc1UKZQ',
} as const; } as const;
export type OperationName = keyof typeof FALLBACK_QUERY_IDS; export type OperationName = keyof typeof FALLBACK_QUERY_IDS;
+44
View File
@@ -313,3 +313,47 @@ export function buildFollowingFeatures(): Record<string, boolean> {
responsive_web_enhance_cards_enabled: false, responsive_web_enhance_cards_enabled: false,
}); });
} }
export function buildExploreFeatures(): Record<string, boolean> {
return applyFeatureOverrides('explore', {
rweb_video_screen_enabled: true,
profile_label_improvements_pcf_label_in_post_enabled: true,
responsive_web_profile_redirect_enabled: true,
rweb_tipjar_consumption_enabled: true,
verified_phone_label_enabled: false,
creator_subscriptions_tweet_preview_api_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: false,
premium_content_api_read_enabled: false,
communities_web_enable_tweet_community_results_fetch: true,
c9s_tweet_anatomy_moderator_badge_enabled: true,
responsive_web_grok_analyze_button_fetch_trends_enabled: true,
responsive_web_grok_analyze_post_followups_enabled: true,
responsive_web_grok_annotations_enabled: true,
responsive_web_jetfuel_frame: true,
responsive_web_grok_share_attachment_enabled: true,
articles_preview_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
view_counts_everywhere_api_enabled: true,
longform_notetweets_consumption_enabled: true,
responsive_web_twitter_article_tweet_consumption_enabled: true,
tweet_awards_web_tipping_enabled: false,
responsive_web_grok_show_grok_translated_post: true,
responsive_web_grok_analysis_button_from_backend: true,
creator_subscriptions_quote_tweet_preview_enabled: false,
freedom_of_speech_not_reach_fetch_enabled: true,
standardized_nudges_misinfo: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
longform_notetweets_rich_text_read_enabled: true,
longform_notetweets_inline_media_enabled: true,
responsive_web_grok_image_annotation_enabled: true,
responsive_web_grok_imagine_annotation_enabled: true,
responsive_web_grok_community_note_auto_translation_is_enabled: true,
responsive_web_enhance_cards_enabled: false,
// Additional features required for ExploreSidebar
post_ctas_fetch_enabled: true,
rweb_video_timestamps_enabled: true,
});
}
+486
View File
@@ -0,0 +1,486 @@
import type { AbstractConstructor, Mixin, TwitterClientBase } from './twitter-client-base.js';
import { TWITTER_API_BASE } from './twitter-client-constants.js';
import { buildExploreFeatures } from './twitter-client-features.js';
import type { SearchResult, TweetData } from './twitter-client-types.js';
const POST_COUNT_REGEX = /[\d.]+[KMB]?\s*posts?/i;
const POST_COUNT_MATCH_REGEX = /([\d.]+)([KMB]?)\s*posts?/i;
/** Options for news fetch methods */
export interface NewsFetchOptions {
/** Include raw GraphQL response in `_raw` field */
includeRaw?: boolean;
/** Also fetch related tweets for each news item */
withTweets?: boolean;
/** Number of tweets to fetch per news item (default: 5) */
tweetsPerItem?: number;
/** Filter to show only AI-curated news items */
aiOnly?: boolean;
}
export interface NewsItem {
id: string;
headline: string;
category?: string;
timeAgo?: string;
postCount?: number;
description?: string;
url?: string;
tweets?: TweetData[];
// biome-ignore lint/suspicious/noExplicitAny: Raw API response can have any structure
_raw?: any;
}
export type NewsResult =
| {
success: true;
items: NewsItem[];
}
| {
success: false;
error: string;
};
export interface TwitterClientNewsMethods {
getNews(count?: number, options?: NewsFetchOptions): Promise<NewsResult>;
}
export function withNews<TBase extends AbstractConstructor<TwitterClientBase>>(
Base: TBase,
): Mixin<TBase, TwitterClientNewsMethods> {
abstract class TwitterClientNews extends Base {
// biome-ignore lint/complexity/noUselessConstructor lint/suspicious/noExplicitAny: TS mixin constructor requirement.
constructor(...args: any[]) {
super(...args);
}
/**
* Fetch news and trending topics from Twitter's Explore page
*/
async getNews(count = 10, options: NewsFetchOptions = {}): Promise<NewsResult> {
const { includeRaw = false, withTweets = false, tweetsPerItem = 5, aiOnly = false } = options;
const debug = process.env.BIRD_DEBUG === '1';
// Try ExplorePage first - this has AI headlines in initialTimeline
if (debug) {
console.error('[getNews] Fetching from ExplorePage (has AI headlines)...');
}
const queryId = await this.getQueryId('ExplorePage');
const features = buildExploreFeatures();
const variables = {
includePromotedContent: true,
withBirdwatchNotes: false,
withCommunity: true,
withSuperFollowsUserFields: true,
withDownvotePerspective: false,
withReactionsMetadata: false,
withReactionsPerspective: false,
withSuperFollowsTweetFields: true,
};
const params = new URLSearchParams({
variables: JSON.stringify(variables),
features: JSON.stringify(features),
});
const url = `${TWITTER_API_BASE}/${queryId}/ExplorePage?${params.toString()}`;
try {
const response = await this.fetchWithTimeout(url, {
method: 'GET',
headers: this.getHeaders(),
});
if (!response.ok) {
const text = await response.text();
return { success: false, error: `HTTP ${response.status}: ${text.slice(0, 200)}` };
}
const data = (await response.json()) as {
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
data?: any;
// biome-ignore lint/suspicious/noExplicitAny: API errors can have any structure
errors?: Array<{ message: string; code?: number; [key: string]: any }>;
};
// Debug: save response if BIRD_DEBUG_JSON is set
if (process.env.BIRD_DEBUG_JSON) {
const fs = await import('node:fs/promises');
const debugPath = process.env.BIRD_DEBUG_JSON.replace('.json', '-explorepage.json');
await fs.writeFile(debugPath, JSON.stringify(data, null, 2)).catch(() => {});
if (debug) {
console.error(`[ExplorePage] Saved response to ${debugPath}`);
}
}
if (data.errors && data.errors.length > 0) {
return { success: false, error: data.errors.map((e) => e.message).join('; ') };
}
const items = this.parseNewsItems(data, count, aiOnly, includeRaw);
if (items.length === 0) {
return { success: false, error: 'No news items found' };
}
if (withTweets) {
await this.enrichWithTweets(items, tweetsPerItem, includeRaw);
}
return { success: true, items };
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
return { success: false, error: `Failed to fetch news: ${errorMessage}` };
}
}
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
private parseNewsItems(data: any, maxCount: number, aiOnly: boolean, includeRaw: boolean): NewsItem[] {
const allItems: NewsItem[] = [];
const seenHeadlines = new Set<string>();
const debug = process.env.BIRD_DEBUG === '1';
if (debug) {
console.error('[ExplorePage] Processing explore_page data...');
}
if (!data.data?.explore_page) {
return [];
}
const explorePage = data.data.explore_page;
if (debug) {
console.error('[ExplorePage] Available sections:', Object.keys(explorePage.body || {}));
}
const body = explorePage.body || {};
// Check the timelines array first (For You, News tabs, etc.)
// biome-ignore lint/suspicious/noExplicitAny: exploring API structure
const timelines = (body as any).timelines || [];
if (debug) {
console.error(`[ExplorePage] Found ${timelines.length} timelines`);
}
for (const timelineObj of timelines) {
if (debug) {
console.error(`[ExplorePage] Timeline ID: ${timelineObj.id}, Label: ${timelineObj.labelText}`);
console.error(`[ExplorePage] Timeline keys:`, Object.keys(timelineObj));
}
const timeline = timelineObj.timeline;
if (timeline) {
const instructions = timeline.timeline?.instructions ?? timeline.instructions ?? [];
if (debug) {
console.error(`[ExplorePage] Timeline ${timelineObj.labelText} has ${instructions.length} instructions`);
}
const itemsFromTimeline = this.extractNewsItemsFromInstructions(
instructions,
timelineObj.__typename || 'timeline',
seenHeadlines,
maxCount,
aiOnly,
includeRaw,
);
if (debug) {
console.error(`[ExplorePage] Timeline found ${itemsFromTimeline.length} AI news items`);
}
allItems.push(...itemsFromTimeline);
if (allItems.length >= maxCount) {
break;
}
}
}
// Also check initialTimeline as fallback
if (allItems.length < maxCount && body.initialTimeline) {
const timeline = body.initialTimeline.timeline;
if (timeline) {
const instructions = timeline.timeline?.instructions ?? [];
const itemsFromInitial = this.extractNewsItemsFromInstructions(
instructions,
'initialTimeline',
seenHeadlines,
maxCount - allItems.length,
aiOnly,
includeRaw,
);
allItems.push(...itemsFromInitial);
}
}
return allItems;
}
private extractNewsItemsFromInstructions(
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
instructions: any[],
source: string,
seenHeadlines: Set<string>,
maxCount: number,
aiOnly: boolean,
includeRaw: boolean,
): NewsItem[] {
const items: NewsItem[] = [];
const debug = process.env.BIRD_DEBUG === '1';
for (const instruction of instructions) {
if (instruction.type !== 'TimelineAddEntries') {
continue;
}
const entries = instruction.entries ?? [];
if (debug) {
console.error(`[${source}] Processing ${entries.length} entries`);
}
for (const entry of entries) {
if (items.length >= maxCount) {
break;
}
if (debug) {
console.error(
`[${source}] Entry ID: ${entry.entryId}, content type: ${entry.content?.__typename || entry.content?.entryType || 'unknown'}`,
);
// Check if this is a "Today's News" or news section header
if (entry.content?.header || entry.content?.displayType === 'VerticalConversation') {
console.error(`[${source}] Found potential news section:`, entry.content?.header);
}
}
const content = entry.content;
if (!content) {
continue;
}
// Handle TimelineTimelineItem (single trend item)
if (content.itemContent && items.length < maxCount) {
if (debug && content.itemContent.is_ai_trend) {
console.error(
`[ExplorePage] Found AI trend in ${entry.entryId}:`,
JSON.stringify(content.itemContent, null, 2).substring(0, 500),
);
}
const newsItem = this.parseNewsItemFromContent(
content.itemContent,
entry.entryId,
source,
seenHeadlines,
aiOnly,
includeRaw,
);
if (newsItem) {
items.push(newsItem);
}
}
// Handle TimelineTimelineModule (multiple items)
const itemsArray = content?.items ?? [];
if (debug && itemsArray.length > 0) {
console.error(`[${source}] Module has ${itemsArray.length} items`);
}
for (const data of itemsArray) {
if (items.length >= maxCount) {
break;
}
// Structure can be data.itemContent OR data.item.itemContent
const itemContent = data?.itemContent || data?.item?.itemContent;
if (!itemContent) {
continue;
}
if (debug) {
console.error(
`[${source}] Module item type: ${itemContent.__typename}, name: ${itemContent.name}, is_ai: ${itemContent.is_ai_trend}`,
);
}
if (debug && itemContent.is_ai_trend) {
console.error(
`[ExplorePage] Found AI trend in module ${entry.entryId}:`,
JSON.stringify(itemContent, null, 2).substring(0, 500),
);
}
const newsItem = this.parseNewsItemFromContent(
itemContent,
entry.entryId,
source,
seenHeadlines,
aiOnly,
includeRaw,
);
if (newsItem) {
items.push(newsItem);
}
}
}
}
return items;
}
private parseNewsItemFromContent(
// biome-ignore lint/suspicious/noExplicitAny: API response structure is complex
itemContent: any,
entryId: string,
source: string,
seenHeadlines: Set<string>,
aiOnly: boolean,
includeRaw: boolean,
): NewsItem | null {
const headline = itemContent.name || itemContent.title;
if (!headline) {
return null;
}
// Detect AI news by characteristics:
// 1. Full sentence headlines (contains spaces and is longer)
// 2. Has social_context with "News" category
// 3. Or explicitly marked as is_ai_trend
const socialContext = itemContent?.social_context?.text || '';
const hasNewsCategory = socialContext.includes('News') || socialContext.includes('hours ago');
const isFullSentence = headline.split(' ').length >= 5; // AI news are full sentences
const isExplicitlyAiTrend = itemContent.is_ai_trend === true;
const isAiNews = isExplicitlyAiTrend || (isFullSentence && hasNewsCategory);
// Filter AI trends if aiOnly is enabled
if (aiOnly && !isAiNews) {
return null;
}
if (seenHeadlines.has(headline)) {
return null;
}
seenHeadlines.add(headline);
let postCount: number | undefined;
let timeAgo: string | undefined;
let category = 'Trending';
// Parse social context for metadata
const socialCtx = itemContent?.social_context;
if (socialCtx?.text) {
const socialContextText = socialCtx.text;
const parts = socialContextText.split('·').map((s: string) => s.trim());
for (const part of parts) {
if (part.includes('ago')) {
timeAgo = part;
} else if (part.match(POST_COUNT_REGEX)) {
const match = part.match(POST_COUNT_MATCH_REGEX);
if (match) {
let num = Number.parseFloat(match[1]);
const suffix = match[2]?.toUpperCase();
if (suffix === 'K') {
num *= 1000;
} else if (suffix === 'M') {
num *= 1_000_000;
} else if (suffix === 'B') {
num *= 1_000_000_000;
}
postCount = Math.round(num);
}
} else {
category = part;
}
}
}
// Parse trend metadata
const trendMetadata = itemContent?.trend_metadata;
if (trendMetadata?.meta_description) {
const metaDesc = trendMetadata.meta_description;
const postMatch = metaDesc.match(POST_COUNT_MATCH_REGEX);
if (postMatch) {
let num = Number.parseFloat(postMatch[1]);
const suffix = postMatch[2]?.toUpperCase();
if (suffix === 'K') {
num *= 1000;
} else if (suffix === 'M') {
num *= 1_000_000;
} else if (suffix === 'B') {
num *= 1_000_000_000;
}
postCount = Math.round(num);
}
}
if (trendMetadata?.domain_context && (category === 'Trending' || category === 'News')) {
category = trendMetadata.domain_context;
}
const item: NewsItem = {
id: entryId || `${source}-${headline}`,
headline,
category: isAiNews ? `AI · ${category}` : category,
timeAgo,
postCount,
description: itemContent.description,
url: itemContent.trend_url?.url || trendMetadata?.url?.url,
};
if (includeRaw) {
item._raw = itemContent;
}
return item;
}
private async enrichWithTweets(items: NewsItem[], tweetsPerItem: number, includeRaw: boolean): Promise<void> {
const debug = process.env.BIRD_DEBUG === '1';
for (const item of items) {
try {
const searchQuery = item.headline;
if (!searchQuery) {
continue;
}
// Use the search method if available (requires search mixin)
if ('search' in this && typeof (this as { search?: unknown }).search === 'function') {
const result = (await (
this as { search: (q: string, c: number, o: { includeRaw: boolean }) => Promise<SearchResult> }
).search(searchQuery, tweetsPerItem, { includeRaw })) as SearchResult;
if (result.success && result.tweets) {
item.tweets = result.tweets;
}
}
} catch {
if (debug) {
console.error('[getNews] Failed to enrich item with tweets:', item.headline);
}
}
}
}
}
return TwitterClientNews;
}
+10 -2
View File
@@ -4,6 +4,7 @@ import { type TwitterClientBookmarkMethods, withBookmarks } from './twitter-clie
import { type TwitterClientHomeMethods, withHome } from './twitter-client-home.js'; import { type TwitterClientHomeMethods, withHome } from './twitter-client-home.js';
import { type TwitterClientListMethods, withLists } from './twitter-client-lists.js'; import { type TwitterClientListMethods, withLists } from './twitter-client-lists.js';
import { type TwitterClientMediaMethods, withMedia } from './twitter-client-media.js'; import { type TwitterClientMediaMethods, withMedia } from './twitter-client-media.js';
import { type TwitterClientNewsMethods, withNews } from './twitter-client-news.js';
import { type TwitterClientPostingMethods, withPosting } from './twitter-client-posting.js'; import { type TwitterClientPostingMethods, withPosting } from './twitter-client-posting.js';
import { type TwitterClientSearchMethods, withSearch } from './twitter-client-search.js'; import { type TwitterClientSearchMethods, withSearch } from './twitter-client-search.js';
import { type TwitterClientTimelineMethods, withTimelines } from './twitter-client-timelines.js'; import { type TwitterClientTimelineMethods, withTimelines } from './twitter-client-timelines.js';
@@ -17,6 +18,7 @@ type TwitterClientInstance = TwitterClientBase &
TwitterClientHomeMethods & TwitterClientHomeMethods &
TwitterClientListMethods & TwitterClientListMethods &
TwitterClientMediaMethods & TwitterClientMediaMethods &
TwitterClientNewsMethods &
TwitterClientPostingMethods & TwitterClientPostingMethods &
TwitterClientSearchMethods & TwitterClientSearchMethods &
TwitterClientTimelineMethods & TwitterClientTimelineMethods &
@@ -25,11 +27,16 @@ type TwitterClientInstance = TwitterClientBase &
TwitterClientUserLookupMethods & TwitterClientUserLookupMethods &
TwitterClientUserTweetsMethods; TwitterClientUserTweetsMethods;
const MixedTwitterClient = withUserTweets( // News mixin wraps search because it depends on the search() method
const MixedTwitterClient = withNews(
withUserTweets(
withUserLookup( withUserLookup(
withUsers( withUsers(
withLists( withLists(
withHome(withTimelines(withSearch(withTweetDetails(withPosting(withBookmarks(withMedia(TwitterClientBase))))))), withHome(
withTimelines(withSearch(withTweetDetails(withPosting(withBookmarks(withMedia(TwitterClientBase)))))),
),
),
), ),
), ),
), ),
@@ -37,6 +44,7 @@ const MixedTwitterClient = withUserTweets(
export class TwitterClient extends MixedTwitterClient {} export class TwitterClient extends MixedTwitterClient {}
export type { NewsFetchOptions, NewsItem, NewsResult } from './twitter-client-news.js';
export type { export type {
BookmarkMutationResult, BookmarkMutationResult,
CurrentUserResult, CurrentUserResult,
+154
View File
@@ -0,0 +1,154 @@
import { Command } from 'commander';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { CliContext } from '../src/cli/shared.js';
import { registerNewsCommand } from '../src/commands/news.js';
describe('news command', () => {
let program: Command;
let mockContext: Partial<CliContext>;
beforeEach(() => {
program = new Command();
mockContext = {
resolveTimeoutFromOptions: () => 30000,
resolveQuoteDepthFromOptions: () => undefined,
resolveCredentialsFromOptions: async () => ({
cookies: {
authToken: 'auth',
ct0: 'ct0',
cookieHeader: 'auth=auth; ct0=ct0',
},
warnings: [],
}),
p: (type: string) => `[${type}] `,
colors: {
accent: (text: string) => text,
command: (text: string) => text,
muted: (text: string) => text,
section: (text: string) => text,
},
l: (key: string) => key,
};
});
it('requires positive count value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--count', '0'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--count must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('rejects negative count value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--count', '-5'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--count must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('rejects non-numeric count value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--count', 'abc'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--count must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('requires positive tweets-per-item value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--tweets-per-item', '0'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--tweets-per-item must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('rejects negative tweets-per-item value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--tweets-per-item', '-3'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--tweets-per-item must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('rejects non-numeric tweets-per-item value', async () => {
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news', '--tweets-per-item', 'xyz'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('--tweets-per-item must be a positive number'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
it('requires both authToken and ct0 credentials', async () => {
mockContext.resolveCredentialsFromOptions = async () => ({
cookies: {
authToken: '',
ct0: '',
cookieHeader: '',
},
warnings: [],
});
registerNewsCommand(program, mockContext as CliContext);
const exitSpy = vi.spyOn(process, 'exit').mockImplementation(((code?: number) => {
throw new Error(`exit ${code}`);
}) as never);
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => undefined);
try {
await expect(program.parseAsync(['node', 'bird', 'news'])).rejects.toThrow('exit 1');
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('Missing required credentials'));
} finally {
exitSpy.mockRestore();
errorSpy.mockRestore();
}
});
});
+360
View File
@@ -0,0 +1,360 @@
import { afterEach, describe, expect, it, vi } from 'vitest';
import { TwitterClient } from '../src/lib/twitter-client.js';
const validCookies = {
authToken: 'test_auth_token',
ct0: 'test_ct0_token',
cookieHeader: 'auth_token=test_auth_token; ct0=test_ct0_token',
source: 'test',
};
type ResponseLike = {
ok: boolean;
status: number;
json: () => Promise<unknown>;
text: () => Promise<string>;
};
const makeResponse = (overrides: Partial<ResponseLike> = {}): ResponseLike => ({
ok: true,
status: 200,
json: async (): Promise<unknown> => ({}),
text: async (): Promise<string> => '',
...overrides,
});
describe('TwitterClient news API coverage', () => {
const originalFetch = global.fetch;
afterEach(() => {
global.fetch = originalFetch;
vi.restoreAllMocks();
});
describe('getNews', () => {
it('returns news items from ExplorePage', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'AI Breakthrough in Machine Learning',
social_context: {
text: 'AI · 2h ago · 15.5K posts',
},
trend_url: {
url: 'https://x.com/hashtag/AI',
},
},
},
],
},
},
],
},
],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(1);
expect(result.success).toBe(true);
expect(result.items).toBeDefined();
expect(result.items?.length).toBe(1);
expect(result.items?.[0].headline).toBe('AI Breakthrough in Machine Learning');
expect(result.items?.[0].category).toBe('AI · AI');
expect(result.items?.[0].timeAgo).toBe('2h ago');
expect(result.items?.[0].postCount).toBe(15500);
expect(result.items?.[0].url).toBe('https://x.com/hashtag/AI');
});
it('filters to AI-only items when aiOnly is true', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'AI News',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: false,
name: 'Regular News',
},
},
],
},
},
],
},
],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(10, { aiOnly: true });
expect(result.success).toBe(true);
expect(result.items?.length).toBe(1);
expect(result.items?.[0].headline).toBe('AI News');
});
it('returns error for non-ok responses', async () => {
const mockFetch = vi
.fn()
.mockResolvedValueOnce(makeResponse({ ok: false, status: 500, text: async () => 'Server error' }));
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(10);
expect(result.success).toBe(false);
expect(result.error).toContain('HTTP 500');
});
it('returns error when API returns errors', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
errors: [{ message: 'Rate limited' }, { message: 'Too many requests' }],
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(10);
expect(result.success).toBe(false);
expect(result.error).toContain('Rate limited');
expect(result.error).toContain('Too many requests');
});
it('returns error when no news items found', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(10);
expect(result.success).toBe(false);
expect(result.error).toContain('No news items found');
});
it('deduplicates headlines', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'Duplicate Headline',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'Duplicate Headline',
},
},
],
},
},
],
},
],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(10);
expect(result.success).toBe(true);
expect(result.items?.length).toBe(1);
expect(result.items?.[0].headline).toBe('Duplicate Headline');
});
it('respects count parameter', async () => {
const mockFetch = vi.fn().mockResolvedValueOnce(
makeResponse({
json: async () => ({
data: {
explore_page: {
body: {
initialTimeline: {
timeline: {
timeline: {
instructions: [
{
type: 'TimelineAddEntries',
entries: [
{
entryId: 'test-entry-1',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 1',
},
},
],
},
},
{
entryId: 'test-entry-2',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 2',
},
},
],
},
},
{
entryId: 'test-entry-3',
content: {
items: [
{
itemContent: {
is_ai_trend: true,
name: 'News 3',
},
},
],
},
},
],
},
],
},
},
},
},
},
},
}),
}),
);
global.fetch = mockFetch as unknown as typeof fetch;
const client = new TwitterClient({ cookies: validCookies });
const result = await client.getNews(2);
expect(result.success).toBe(true);
expect(result.items?.length).toBe(2);
});
});
});