Separate "debug" behavior from "billable" behavior

[dbemiller]

Motivation

One of our consultants surfaced the following publisher workflow. According to him, it's quite common.

1. Make a dev/test environment, and use Prebid.js debug mode to verify it.
2. Release these changes to prod, keeping debug mode enabled.
3. Use debug info & analytics data to make sure that prod is working.
   4? Maybe remove debug mode once everything's working.

To be clear, publishers should do (4). Debug info makes the PBS Response huge, which will have a major impact on site performance.

But during that transition period, it's still reasonable to want debug info and billable impressions.

Enter: our current predicament. In Prebid Server today, test is synonymous with debug. And the OpenRTB spec defines bidrequest.test to be:

Indicator of test mode in which auctions are not billable, where 0 = live mode, 1 = test mode.

It looks like I made a bad choice, here. Our current API doesn't support Publishers during this transition period, when they want debug info and billable impressions.

Feature Request

Add support for a boolean request.ext.prebid.debug flag. If present, we should send all the same debug info that we do today when request.test == 1.

For backwards compatibility, we will still need to send debug info when request.test is 1 as well.

After completion

When this ticket closes, we'll need to open two new issues:

1. To Prebid.js, change the prebidServerBidAdapter so that it sends request.ext.prebid.debug rather than request.test when in debug mode.

2. To Prebid Server: In the future, stop responding with debug info when "test": 1. Once the debug flag exists, this behavior will be confusing.

[bretg]

This is done for PBS-Java. @hhhjort - let me know when PBS-Go is ready. Then we can update the PBJS PBSbidadapter to send ext.prebid.debug instead of test when in debug mode. And we may add a ?pbjs_test to go along with ?pbjs_debug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[laurb9]

If the publisher does not turn the flag off then this can skew timing analytics, trigger unusual pattern alerts and increase server egress costs.

Maybe there should be a way to rate-limit such requests or control them server-side.

[bretg]

That's a good point @laurb9 , but if the debug or test flag is rate-limited, how will you tell which requests really need it? i.e. if we added a "sample" flag to say "only actually debug on 10% of debug requests", then testers who really want the debug would have to hit it 10 times to get one.

While I agree that leaving debugging on for all site traffic for any length of time is a bad idea, I lean towards thinking this is a documentation and operational problem, not something to configure.

[SyntaxNode]

I agree with @bretg on not building this in to PBS.

Rate limiting is also hard to get right. Most (all?) hosts have multiple servers and would need infrastructure to coordinate between servers on rate limiting stats. PBC has a naive approach built in, but it uses a lot of memory and is isolated to each machine, so it's not even very effective.

@laurb9 How would you feel if we add metrics to track per-account the number of requests received that requested debug information as either the test or debug flags? That'll at least give the host companies operational information to potentially act on.

[laurb9]

@SyntaxNode that should be fine, it could help correlate with observed increased latency. We don't have a lot of account metrics.

I think if and when #1426 makes it in and we have account configurations, I should be able to turn it off at account level if necessary. It is not a priority by any means, I only want to point out that possibility exists for accidental abuse in an environment where otherwise corrective action relies on third party so could be slow.

[SyntaxNode]

I'm closing this issue since the functionality is available in both PBS-Go and PBS-Java.

@laurb9 I created a new issue to for debug metrics. If there's any other feature request suggestions you'd like to share, please open them as new issues.