There appears to be a consensus that unstructured interviews are kind of a waste of time, mainly because various studies have shown that people are quite bad at assessing relevant candidate characteristics. To cherry-pick a few examples:

University of Texas Medical School applicants who were initially rejected by an interview process - but then accepted due to an increase in the size of the class - performed no worse than those students accepted the first time around.
Face to face auditions for symphony orchestra candidates led to discrimination against women. The advancement probability for female applicants rose by 50% once initial auditions were instead conducted behind a blind screen.
In an experiment, interviewers came up with confident impressions of candidates even when the interviewee had been directed to provide random responses.

It's hard to find a paper that says unstructured interviews are more useful than a monkey throwing darts. However, my own semi-recent interviewing experiences have seen a major emphasis placed on these kinds of ambiguous assessments. In an attempt to justify an economics background, I find myself really wanting to believe that firms are rational; surely massive corporations know that having a candidate's future decided after fifteen seconds is sub-optimal? So, er, why do these interviews exist?

I. Hiring Is Not Quantitative

The first thing to note is that the academic literature generally focuses on connecting interviews to measurable outcomes such as student GPA or dropout rate. Unfortunately, these success metrics are tough to define at an employee-level for most businesses - especially as a firm grows larger - and even then it can often take years to validate a hire with any confidence.

Realistically, I think most businesses are satisfied with three traits in a technical applicant:

Trait 1. Must be reasonably competent, at least above some set threshold.
Trait 2. Must be reasonably competent, without driving coworkers up the wall.
Trait 3. Must be reasonably competent, and stay with the firm for a reasonable length of time, in order to recoup costs associated with hiring, training etc.

Being a quantitative metric all-star is usually not a requirement.

Trait 2 likely explains why many corporations persist with unstructured "get to know the candidate" interviews; firms understand that the negative externality of hiring a team member everyone hates vastly outweighs the slight increase in expected proficiency gained by switching to a more technical interview format.

II. Let's Generalize

Justifying unstructured interviews does not mean that the typical corporate interview process is perfect. It's probably still terrible. And I do think there are some fixes to be made based on certain assumptions:

Assumption 1. You do NOT know who on your team is a good interviewer:

Let's say you have a bad egg. This individual is able to persuasively argue for or against any candidate, often convincing other team members who may have been on the fence. Unfortunately, the same person is completely wrong in the long-run on every single applicant.

How would you mitigate their influence? In my mind, there are some easy solutions:

Randomized assignments for interviews. No one interviews every candidate, not even management. No one is allowed to self-select into interviewing a particular applicant.
Interviewers give anonymized, independent feedback on a candidate, without sharing their thoughts prior to submittal.
In the long-run, track interviewer feedback and compare it with measurable outcomes like job tenure, promotion decisions etc. This is far from perfect, but should catch the worst of the worst when it comes to interviewer judgment.

Assumption 2. The Internet will find out your secrets:

If a major firm uses the same questions consistently, these questions will leak. People talk, and some post to Glassdoor. Interview questions therefore need to be robust to a candidate likely knowing them ahead of time.

Assumption 3. Being normal is easy in the short-run, hard in the long-run:

Team chemistry matters, so discerning a candidate's personality is an important part of the interview process. In my mind it's the only valid reason for allowing an unstructured interview.

Of course, a 30 minute interview in a drab conference room is probably the worst possible way to discover someone's true personality outside of having them create a dating profile. To figure out what someone is actually like requires time.

Assumption 4. Your company's absolute worst case scenario should be someone who is pretty decent at the job:

A sociopath can play nice and you might never detect it even if you interviewed them for weeks. Likewise, someone who is planning to jump ship at the first sign of trouble can come off as a dedicated team player. But your company has absolutely no excuse for hiring someone who doesn't have the technical ability to do the job. Any interviewing strategy should ensure that Trait 1 is met.

Assumption 5. Hiring process improvements must not come with a considerable increase in time commitment:

Nobody wants to spend all day conducting phone screens, interviewing candidates on-site, and filling out long-winded interview feedback forms. So any changes need to fit within a typical applicant loop of 1-2 phone screens followed by an on-site. Advancement probabilities at each stage also need to remain approximately constant to avoid necessitating many more candidates in the pipeline.

III. Putting It All Together: The Walker Process For Slightly Better Technical Hiring Than Your Company Has Right Now (TWPFSBTHTYCHRN)

Prologue. Resumé review followed by an initial phone screen with a recruiter:

I know very little about how these stages work and won't pretend to have any advice on improving them. Let's assume a candidate has cleared early hurdles and is now ready to be interviewed by someone on the team to which they would be hired.

Step 1. Video Screen:

Conducted by a randomly assigned team member. Not a phone screen, as we need to know it's you and not your roommate. Ask rapid-fire questions on technical topics relevant to the role. Since even the best candidate can trip over a single question, run through many to reduce variance. Questioning en masse also makes it tougher for a candidate to have prepped for everything ahead of time.

Step 2. Video Screen, Feedback:

Completely quantitative. A candidate correctly answering 8/10 questions is preferred to another that only got seven. No personality judgment needed.

Step 3. On-Site, Prep:

We spend a reasonable budget to make sure the candidate has a nice experience traveling in. Given that relaxed candidates likely do better we want the applicant in a good frame of mind. Plus it might make them more inclined to accept an offer. The candidate is informed that most of the interview will consist of a case study, and to list any software preferences (e.g. access to RStudio, Jupyter notebooks).

Step 4. On-Site, Morning:

We welcome the applicant with a nice breakfast and lots of coffee (the interview is supposed to be tough, not cruel and unusual punishment). The case study is introduced. The candidate is given a computer to work on the case with appropriate software, including whatever was requested. Oh, and there's internet access. Every analyst uses Stack Overflow daily- why would we prevent this in an interview setting?

Sidenote 1: The construction of the case is very important. It should ask a typical business question using a typical dataset, albeit simplified to fit the time constraints of a one day on-site. If a team finds that creating a case study relevant for a role is difficult, this is a huge red flag! Maybe the team is out of its depth hiring for this position and should look for external help?

The candidate works on the case all morning with minimal supervision. Any candidate stupid enough to try to cheat, such as by contacting a friend over the company's network, should be easily caught. Candidates are mainly evaluated through showing their work, anyway, so an applicant would have to download more than just the correct answer.

Step 5. On-Site, Lunch:

Three randomly assigned team members are chosen to take the applicant out. It's a LONG lunch. We want the candidate to let their guard down. If someone asked you to play a character for thirty minutes do you think you could manage it? Probably. For two hours? A little trickier. For some reason candidates often seem to forget that an interview doesn't stop when lunching. The lunch also gives the individual a chance to ask questions about the role and the company.

Sidenote 2: As much as I want a technical interview to only include technical questions, including this step is a concession that firms will always want to judge "culture" fit. Unfortunately such a criterion makes discrimination quite easy, not only based on race and gender, but also based on things like educational background and previous work experience. A compromise allows judgments on personality while relegating them to a minor part of a comprehensive candidate evaluation. I also don't have an issue with introducing interview quotas for minority classes within gender, race, and other protected categories, as well as for non-Ivy Leaguers in Silicon Valley.

Step 6. On-Site, Afternoon:

After returning from lunch the candidate finishes up the case study and presents findings to a different subset of the team than the group that took the applicant out to lunch. The results presented are pretty irrelevant, especially if the candidate has little background in the industry, since the goal here is just to demonstrate an ability to showcase work clearly.

A candidate's code is also checked...failing to write a decent SQL query would be sub-optimal. Note that even if a candidate is completely aware of the case study prior to interviewing, the majority of the evaluation still rests on their ability to write clear, correct code and to present findings in an intelligible manner. This is much harder to prep for than just memorizing the answers to some arbitrary logic problems.

Stock images place a heavy emphasis on including pie charts in presentations. I do not recommend this.

Step 7. On-Site, Feedback:

Evaluations are submitted completely independently by each team member. Offer decisions are given based on averaging quantitative candidate evaluation scores, with the case study component weighted more highly than the lunch or the original video screen.

Overall, the interview has been designed to test what a candidate actually knows, and whether that knowledge is relevant to the particular role. The ability to take credit for past projects should have no relevance, and flimsy personality judgments are given relatively minor weight (without removing completely!). A deep-dive case study prevents candidates from memorizing answers to leaked questions. The whole process shouldn't require any more of a time investment from the company's perspective than a typical unstructured interview loop, while independent feedback forms and randomized interview assignments are used to mitigate any single team member's influence. From a candidate's viewpoint, the worse case on-site scenario is a nice breakfast and an excellent long lunch: so surely everyone wins?