Summary:
In this episode of Sidecar Sync, Mallory Mejias welcomes Dr. Jessica Miles, founder of The Informed Frontier, former VP of Strategy and Investments at Holtzbrinck, and advisory board member for Johns Hopkins University Press, for a fascinating look at how AI is transforming publishing, research, and trusted content ecosystems. Jessica explains what happens when AI becomes a “reader” instead of just a writer or assistant, why associations and scientific societies may already be part of AI training pipelines, and how organizations should think about crawl-based access, bulk licensing, and runtime access as the emerging gold standard. The conversation also explores AI’s impact on research integrity, peer review, content metrics, and the irreplaceable role of human accountability in an AI-driven knowledge ecosystem.
🔍 More about Dr. Jessica Miles:
https://www.theinformedfrontier.com/
https://www.linkedin.com/in/jessicalmiles/
https://podcasts.apple.com/us/podcast/the-informed-frontier/id1842407705
Timestamps:
00:00 - Meet Dr. Jessica Miles
👥Provide comprehensive AI education for your team
https://learn.sidecar.ai/teams
📅 Register for digitalNow 2026:
https://digitalnow.sidecar.ai/digitalnow
🤖 Join the AI Mastermind:
https://sidecar.ai/association-ai-mas...
🎀 Use code AIPOD50 for $50 off your Association AI Professional (AAiP) certification
📕 Download ‘Ascend 3rd Edition: Unlocking the Power of AI for Associations’ for FREE
OpenAI ➔ https://openai.com
ChatGPT ➔ https://chatgpt.com
Cloudflare ➔ https://www.cloudflare.com
ProRata ➔ https://www.prorata.ai
VetGeni ➔ https://www.vetgeni.com
OpenEvidence ➔ https://www.openevidence.com
DataSeer ➔ https://dataseer.ai
https://www.linkedin.com/company/sidecar-global
https://twitter.com/sidecarglobal
https://www.youtube.com/@SidecarSync
⚙️ Other Resources from Sidecar:
More about Your Hosts:
Amith Nagarajan is the Chairman of Blue Cypress 🔗 https://BlueCypress.io, a family of purpose-driven companies and proud practitioners of Conscious Capitalism. The Blue Cypress companies focus on helping associations, non-profits, and other purpose-driven organizations achieve long-term success. Amith is also an active early-stage investor in B2B SaaS companies. He’s had the good fortune of nearly three decades of success as an entrepreneur and enjoys helping others in their journey.
📣 Follow Amith on LinkedIn:
https://linkedin.com/amithnagarajan
Mallory Mejias is passionate about creating opportunities for association professionals to learn, grow, and better serve their members using artificial intelligence. She enjoys blending creativity and innovation to produce fresh, meaningful content for the association space.
📣 Follow Mallory on Linkedin:
https://linkedin.com/mallorymejias
🤖 Please note this transcript was generated using (you guessed it) AI, so please excuse any errors 🤖
[00:00:00:14 - 00:00:09:17]
Mallory
Welcome to the Sidecar Sync Podcast, your home for all things innovation, artificial intelligence and associations.
[00:00:09:17 - 00:00:32:05]
Mallory
We spend a lot of time on this podcast talking about AI as a writer, an assistant or a creator. But what happens when AI becomes a reader?
[00:00:33:07 - 00:01:20:15]
Mallory
Today's guest, Dr. Jessica Miles has spent years thinking about that question from inside the worlds of scientific publishing, research and AI governance. She's the founder of the Informed Frontier, former VP of Strategy and Investments at Holtz Brink and currently serves on the advisory board of Johns Hopkins University Press. And in this conversation, we explore what happens when AI changes not just how content is written, but how it's discovered, licensed, summarized and consumed. In this episode, we cover why associations and publishers may already be training AI models, whether they realize it or not, the difference between crawl based access, licensing deals and the gold standard emerging for AI content access.
[00:01:21:18 - 00:01:40:09]
Mallory
Why associations may become highly cited by AI systems without anyone ever visiting their websites, how AI is impacting research integrity, peer review and scientific publishing. And why Jessica believes only humans can be accountable for outcomes in an AI driven knowledge ecosystem.
[00:01:41:14 - 00:01:52:11]
Mallory
If your organization sits on decades of trusted content and expertise, which I imagine it does, this episode is a really important listen. Please enjoy this interview with Dr. Jessica Miles.
[00:01:52:11 - 00:02:07:00]
Mallory
Jessica Miles, thank you so much for joining us on the Sidecar Sync podcast. I've got to say it's always refreshing when I have a fellow podcaster on the podcast because I know that you just get it. So thank you for joining us.
[00:02:07:00 - 00:02:12:12]
Dr. Jessica Miles
I'm happy to be here and you flatter me. So I'm looking forward to a great discussion.
[00:02:13:13 - 00:02:34:15]
Mallory
Absolutely. Many people who debate the future, I think of AI and publishing, they might come from a tech background or a product background. You came in through microbiology, a PhD from Yale. So I'm hoping you can tell us a little bit about your background and how that set you up in this moment to speak on the very nuanced topic of AI and publishing.
[00:02:35:24 - 00:03:37:11]
Dr. Jessica Miles
Absolutely. And it's been a fascinating road so far. Certainly I didn't envision talking about AI supporting societies when I was studying microbiology, but it's been a really gratifying path. I founded the Informed Frontier, my advisory firm to support organizations navigating AI's impact on the way we access, interpret and share information. And as you mentioned, I started my journey as a PhD. So I'm a trained scientist and it's really been a natural fit for me to support societies and associations that are really on the forefront of science, technology and innovation. Shortly before founding the firm, I was a VP of strategy and investments at Holtzbrink, where I really was front and center to seeing the impact that generative AI had on science publishing, ed tech and media. And in that role, I oversaw strategic planning as well as acquisitions and investments in AI power tools and helped shape responsible AI policy. So I got to draw on a lot of really interesting and fun experiences when I founded my own firm.
[00:03:38:17 - 00:04:21:23]
Mallory
Absolutely. In my research on you, Dr. Miles, for this episode, I had to dig deep because I'm not super familiar with the area of publishing at large. And as I mentioned to you before we started this recording, we haven't really talked about AI in publishing too much on this podcast. So I did find a debate that you participated in. I believe it was a keynote for the Society for Scholarly Publishing. And you had to debate whether AI would fatally undermine the integrity of scholarly publishing. You argued no, and the audience disagreed. And I thought if we could revisit that for a moment, have your views changed? Is there anything you feel like needs to be added to that conversation? I feel like it would really set the tone for this conversation.
[00:04:22:24 - 00:05:01:08]
Dr. Jessica Miles
I'm so happy that you found that, even though we are revisiting my loss in the debate. But to frame the context, as you said, this was a keynote for the Society for Scholarly Publishing, 2023. And the premise was an Oxford style debate, which involves setting a premise, as you said, AI will fatally undermine the integrity of scholarly publishing. And then each debater either argues for the proposition or against the proposition. And so I was assigned to argue against the proposition. And I highlight that because my debate partner, Tim Vines, who's the founder of Data Seer,
[00:05:02:10 - 00:05:34:14]
Dr. Jessica Miles
is also I think actually a little bit more moderate in his views than he espoused that day as am I. Being in that debate kind of really forces you to take quite an extreme position. Sure. But I look back on that because I got to argue that no, AI is not going to essentially destroy publishing. I look back with a lot of pride on that experience because some of the things that I highlighted have really started to come to fruition in terms of AI making publications more accessible, more discoverable, more usable.
[00:05:35:17 - 00:06:27:02]
Dr. Jessica Miles
And seeing that publishers are using AI tools to continue to innovate on research integrity, as I said, on discoverability, on supporting scientific research. And so I feel like a little bit like some of those seeds that were kind of planted in 2023 started to blossom when you look at AI in terms of streamlining peer review, or making knowledge in science more widely disseminated, or even supporting publishers in terms of marketing and improving the user experience. So again, I look back on that experience really fondly. I think if anything, the sentiment has shifted in my favor in terms of people kind of really accept that these AI tools are ubiquitous, they're here to stay. And the question becomes, how do we navigate this changed and continually changing world?
[00:06:28:17 - 00:07:05:06]
Mallory
I've got to commend you on that too. I had to do a debate, I think in a college Spanish class. And being assigned a side of a debate can be quite intimidating, when maybe even if you have different personal views, you have to really double down and say, all right, these are the facts I'm presenting. You mentioned the discoverability of scientific information and the wide dissemination of scientific knowledge. I want to go into that segment for this episode. So season one of your podcast, The Informed Frontier focuses on AI as a reader of content. Why frame it in that way instead of focusing on AI as a writer or an assistant the way many might?
[00:07:06:11 - 00:07:41:18]
Dr. Jessica Miles
That's a great question. I think part of it came out of really just seeing this information access piece and the way AI was sort of really upending the paradigms that publishers and societies have been navigating under for 100 plus years. Everything that we've done in scientific publishing, the infrastructure we've built, the workflows, all of it has been predicated on the assumption that a human, whether that's a researcher, a policymaker, or an educator, is the one that's accessing content, whether that's journals, books,
[00:07:42:18 - 00:08:20:18]
Dr. Jessica Miles
standards, other materials. And that person is the one making the decision of what to read and what not to read. And of course, with AI, that's shifted. Obviously we still have human readers that need to be served capably. But now we have these emerging so-called AI readers. And so the question becomes, how do you maintain what we might call these legacy systems that support human readers that we very much still need to support while building infrastructure that enables AI to really break down barriers, push boundaries, and really generate this next wave of innovation?
[00:08:22:12 - 00:08:58:15]
Mallory
We've talked about that specific piece, AI readers on the pod with the, I won't say recent, but perhaps in the last year or so, the Cloudflare announcement that you could block AI crawlers or pay-per-crawl. And we've discussed that a bit on the podcast, the idea of, well, you could block down all your content. However, you kind of risk not being discovered in that sense. So I'm curious, what is your take on balancing keeping your content under lock and key, which you rightfully should, or getting paid when that content is accessed versus making sure as an association, a small association, especially you're still out there to be discovered? What do you think about that balance?
[00:08:58:15 - 00:10:05:21]
Dr. Jessica Miles
No, I think balance is a really operative word in terms of you don't want a situation where no one can access your content because that's counter to the mission of disseminating knowledge. However, you do want some friction in terms of pausing these AI bots a little bit. And one of the guests I had on the podcast referred to it as, quote unquote, "closing the back door." The idea being that if you help prevent unmitigated access to your content, then those folks who are seeking your content, whether those are frontier model builders or people using AIs for other applications, even librarians, researchers, they have to come to the front door and speak with you directly. And another way I've heard this frame is thinking about three different modes of AI access. The first is crawl-based access, which you can think of as kind of that first scenario where again, you're getting unmitigated access on these AIs or sometimes we use the word "scraping" publicly available content and using that either for training or real-time retrieval methods such as inference.
[00:10:06:23 - 00:10:51:01]
Dr. Jessica Miles
The second method is bulk ingest. And here's where the tables kind of turn a little bit. So bulk ingest involves a publisher or an information provider giving access to that content upfront. And then an AI is able to ingest that in a kind of single use manner, hence the bulk part, and then refer to that in terms of retrieval or other activities. And so bulk ingest requires licensing, which is a plus for publishers. And so it gives you a little bit more control over how that content is used. The downside is though, in terms of measuring that usage, you're still entirely dependent on the application. So if I, Jessica Miles,
[00:10:52:05 - 00:11:16:12]
Dr. Jessica Miles
license content to, we'll just use OpenAI as an example through this bulk ingest kind of mode, I entered into a license with them, which is great. They're not kind of scraping my content, but I also have to rely on them for how that content is used in terms of training their models, in terms of the outputs. I don't really have any direct kind of line of sight into that.
[00:11:17:13 - 00:11:22:04]
Dr. Jessica Miles
The third method, which isn't even an improvement on this kind of bulk ingest,
[00:11:23:09 - 00:11:35:10]
Dr. Jessica Miles
you can think of as runtime access. And so AI tools are able to access content via an authenticated access method, like an API. And they're using that to generate the response.
[00:11:36:10 - 00:12:04:00]
Dr. Jessica Miles
And so this is really an ideal scenario for content holders like societies or publishers, because they're able to not only control how that content is accessed by the AI, but they also still maintain that transparency into how that content is being used in the outputs. So that seems like all upside, but the challenge is really that it's obviously time consuming to set up these sort of authenticated modes of access.
[00:12:05:12 - 00:12:41:03]
Dr. Jessica Miles
There's also the expense and operational costs involved in maintaining them. But if you're looking for kind of the gold standard of, okay, I not only want to control who's using my content, but I want to have more transparency into how that content is being used. You know, runtime access is really, I think, where the industry is starting to shift in terms of, I don't want to say demanding greater transparency, but asking and increasingly getting that transparency, which is obviously so fundamental to research academic scholarship. And so it's really nice that we're starting to see that enable these technologies as well.
[00:12:42:18 - 00:13:01:00]
Mallory
That's really helpful. I imagine there are some listeners thinking, well, most of our good vetted quality content is behind a paywall. So we are safe or behind a member login. Would you agree with that statement? What are you seeing on your side with crawlers, perhaps accessing content that's behind a paywall? Is that possible?
[00:13:02:04 - 00:14:39:15]
Dr. Jessica Miles
Yes, is the short answer. I think the long answer is you see kind of a couple modes whereby crawlers are accessing that content. The first is through accessing sort of known corridors, if you will, of paywall content for a lot of, you know, STM publishers, site, Technical Medical. We know that content has been acquired through illegitimate and often legal means. And if it's publicly available, then it's publicly available. So it gets crawled. The second is through security vulnerabilities. So the way that access is often authenticated by institutions and etc. is a little bit of a legacy method in terms of IP usage, which is not common in other sectors, but it's very common with academic institutions, for example, because it allows a lot of users to use that particular account to access content. However, you do see situations where IP addresses can be proxied or faked, if you will, which can allow sort of these malevolent actors to access paywall content. And so that's probably less common than kind of scraping publicly available pirated content. But the end result is the same in terms of paywall content is not necessarily safe. But because again, I think for a lot of societies and publishers, paywall content holds a unique value. It's important to be thinking about systems for safeguarding publicly available or open access content, as well as paywall content.
[00:14:40:21 - 00:15:10:02]
Mallory
So it sounds like the three options that you've discussed so far for having your for being discoverable in terms of your content or crawl based access, which is kind of like the low, we'll call it the lowest tier, perhaps the most least desirable. You've got bulk ingest through licensing and then kind of the gold standard that you mentioned is runtime access. Are you seeing only large scientific organizations and societies go with the runtime access route? Do you think it's something feasible for smaller organizations too?
[00:15:10:02 - 00:16:05:22]
Dr. Jessica Miles
I will say it's rarer. I think in general, the larger organizations that have the most content have been a little bit more at the forefront and more willing to invest in these technologies. That being said, I think even if you're at the point where you're pursuing licensing arrangements and having bulk ingest, that's a huge win. Not everyone has been able to successfully implement these licenses and control that access. So that's a really critical first step. But in terms of getting to that third tier, there are technology providers that work with smaller organizations. And so figuring out what your corpus looks like, what sorts of organizations you would want to work with and really having a strategy in place for thinking about how to just submit your content, I think it is possible for smaller publishers. It just maybe looks a little bit different on paper.
[00:16:07:18 - 00:16:39:23]
Mallory
I want to spend a little bit more time talking about licensing because I was hoping to dedicate kind of a portion of this episode to that. So I did some digging and I want to provide some context to our listeners first. Publishers are obviously making some serious money from AI licensing. Wiley, for example, has earned over $50 million. News Corp has a $250 plus million deal with OpenAI. Trade associations, very relevant for our listeners as well, are getting into the mix. The News Media Alliance brokered a collective deal with the AI company ProRata for its 2,200 members.
[00:16:41:00 - 00:16:54:10]
Mallory
So Jessica, can you help set the table for us? If an AI company comes knocking with a licensing offer, what is actually on the table? What do scientific societies or associations in general need to be looking out for?
[00:16:55:14 - 00:18:01:00]
Dr. Jessica Miles
I'm very glad that you threw out kind of all that supporting data to help frame this discussion because it kind of allows me to draw a little bit of the arc that's transpired from kind of the early days of these licensing deals to where we find ourselves now. So you mentioned Wiley having kind of enormous success and a lot of revenue associated with these licensing deals. And one kind of asterisk I would put with that is that a lot of that came from licensing its corpus for model training, which is distinct activity from some of the, like I said, real-time uses of AI like inference, RAG, grounding. You hear a lot of terms, but I'll just sort of bucket them as real-time access for now. The training deals tended to be higher revenue. And they also tended to be the domain of large publishers because a lot of, especially, I said a lot of AI companies, especially the frontier model companies, you know, were thought, "Okay, who can I target with a large corpus in order to get the most kind of bang for my buck in terms of just training these models?"
[00:18:02:01 - 00:18:34:05]
Dr. Jessica Miles
So on the one hand, it might have been less available to smaller organizations. But on the other hand, we're seeing that as the years go by, the emphasis is less on these sort of large value training deals and more on inference and other real-time uses, which come with smaller payouts, but more frequent payouts because there might be a training is generally a kind of a single event or at least single event in the context of the agreement.
[00:18:35:08 - 00:20:57:15]
Dr. Jessica Miles
But things like runtime access or things where the models are having to continually use and service the content, those enable opportunities for generating payments or revenue on a recurring basis. And so that's sort of, I would say, opportunity or benefit over training. And it also means that there is a focus on potentially more specialized content. Because if you think about it, if you're just training a model, you know, you're kind of looking, as I said, for a large corpus, sort of broad information. But if you need to train a model to do something very specific, then you place a premium on content information that relates to that use case. So we'll use Wiley again, just because, A, you know, you talked about them, and B, they've been very public about their activities. So they have a partnership with a specialized application provider called VetGenie, where they've licensed a lot of their veterinary content for this app that's specifically for veterinarians, right? So you might know the association space better than I do in terms of a society that had a specific corpus related to veterinary content, for example, that might be valued by somebody like a VetGenie who's building an application for veterinarians. We've seen other examples in the medical space, for example, Open Evidence, which is an application for medical and clinical providers reaching out to the New England Journal of Medicine, which is actually a society. They're headquartered here in Massachusetts. It's the Massachusetts Medical Society, as well as, you know, AMA, the American Medical Association licensing their content with with JAMA and those other journals. So I think the arc is kind of too full. We're looking at changes in the type of activities that AI companies are doing with content. We're seeing changes in the payment plans or the payment frequency, if you like, associated with those activities. But we're also seeing changes in the content that's valued in terms of accompanying those two changes from this sort of broad scope corpus that's used for training versus more specialized domain knowledge that is used for either real time access or building application specific uses.
[00:20:58:17 - 00:21:32:11]
Mallory
I realize many scientific societies don't actually run their own publishing operations. Some do, but they partner with one of the bigger publishers like Wiley, for example, if we keep going back to them, not picking on Wiley, for the sales, the library contracts. So when those publishers cut AI licensing deals, the society's content is part of what's being licensed, but the society isn't the one at the negotiating table. So from your vantage point, have you worked having worked inside the publisher side? How do you think societies should be thinking about that if they do work with big publishers who are making these types of deals?
[00:21:33:21 - 00:22:58:09]
Dr. Jessica Miles
That's a great question. And that's I'll also highlight that's a trend that we're seeing increasingly that societies are partnering with these larger commercial, generally, but not always commercial, but certainly larger publishers in terms of having partnerships around the production and dissemination of their content. You mentioned that when these publishers are entering into deals, sites aren't directly at the negotiating table. I would say it depends. So I know from my experience as a publisher at a large commercial publisher, yeah, there was too many uses of publishers, but I manage a portfolio of publications for a society at a large publisher. We often would exclude societies from certain activities, whether that was more internal operational. And I'm sure with these licensing deals, it very much depends on the society and the arrangement as to whether or not societies are included. So I would say the first step, if you are a society being published by a larger commercial publisher, is to ask for transparency in terms of how that's negotiated, especially because some of these partnerships between larger publishers and societies operate under agreements that may be in existence for seven, 10 years. Oftentimes, if you're operating under something that was drafted in 2020 or even before that, this type of licensing wasn't really contemplated.
[00:22:59:11 - 00:23:09:04]
Dr. Jessica Miles
So it might not be specifically covered in terms of who owns the copyright, who is licensing to whom, what activities the publisher is allowed to undertake.
[00:23:10:23 - 00:23:48:19]
Dr. Jessica Miles
A lot of that is going to vary contract by contract. And so I think it's really, really the first step is to have that conversation. If you want to be opted into these deals wholesale, certainly I think publishers would be really excited if their society partners are really looking for that visibility. That said, some societies really do want to retain a right to advocate for themselves and negotiate more specifically on their own. And so those conversations are a little bit more nuanced, but certainly need to be had. So I would say, I don't assume that you know whether you're included.
[00:23:49:20 - 00:23:52:10]
Dr. Jessica Miles
Talk to your publisher early and often.
[00:23:53:16 - 00:23:59:23]
Mallory
It sounds like you're saying make sure that you have a seat at that table or at least you know what's going on in that room. Absolutely, absolutely.
[00:24:01:14 - 00:24:48:05]
Mallory
Something I want to go back to that we kind of talked about at the top of this conversation was your debate. And again, in my research for this episode, I realized that AI is impacting publishing kind of on the front end and the back end. On the front end, in terms of perhaps paper mills, like generating the actual papers themselves. On the back end, having AI summarize the paper so people just read the summaries if a human's involved in that process at all, as opposed to reading the whole paper. So I'm curious from that angle, going back to your debate, it sounds like in that process, AI could certainly be undermining the art of publishing in general. So can you speak a little bit about that kind of AI's impact on the front end and the back end and how we can mitigate that as humans?
[00:24:49:15 - 00:25:04:11]
Dr. Jessica Miles
Gosh, that's such a big question. So I'll maybe start on the front end. I think when we think about research integrity, those conversations have been ongoing for some time in terms of thinking about reproducibility. That's been a large technical conversation.
[00:25:05:11 - 00:25:36:20]
Dr. Jessica Miles
And in some ways, AI, for example, makes it easier to commit research fraud, but also helps research to improve research reproducibility in terms of being able to do things like run code more easily during the review process. So you have kind of a balance of activities there. I think a lot of the research fraud that has been seen in scientific publishing, certainly pre-dates generative AI, I think generative AI makes it easier to, for example, do things like image manipulation.
[00:25:38:08 - 00:27:08:11]
Dr. Jessica Miles
But on the other hand, it also gives us these tools to review and assess papers more easily and at scale. So it used to be, it was pretty challenging to have these algorithms that would detect image manipulation or detect text or fraudulent text or certainly we've had plagiarism checkers for decades now. But it's a bit of a balance because you have, as you mentioned, AI perhaps amplifying some challenges that existed previously with respect to fraud and research integrity. But also providing tools to combat it. So it is a little bit of an arms race. I think at the moment and who's winning at any given point, it can be difficult to say. I think what we are starting to see come online are better ways of kind of verifying whether images or text are AI generated. And so again, if that kind of information is disclosed and allowable in terms of the regulations or policies of the journal, that in and of itself is not necessarily a bad thing where it gets to be problematic, of course, if it's used in a fraudulent way or to service fraudulent ends. And so you mentioned kind of on the back end, but I might kind of go to the middle because obviously peer review is such a critical part of research.
[00:27:09:18 - 00:28:12:01]
Dr. Jessica Miles
And once again, even before we were using these generative AI systems at scale, peer review was under strain a little bit. The amount of research being published each year increases, but generally there aren't any more kind of peer review, or at least not the equivalent number of more peer reviewers to review the research. So there's been this challenge of who is going to be able to review all of this new research that's coming out. And so in some ways, AI is a boon in terms of providing tools again to expedite the review process. But just like on the front end, it has to be used appropriately. It should not be used in place of a peer reviewer. It should be used as a tool to make a peer reviewer's job easier. So this is becoming a theme where AI can perhaps exacerbate existing problems. It can offer solutions, but how it's used is so critically important in terms of tipping the scales towards one end of the spectrum or the other.
[00:28:13:06 - 00:28:29:05]
Mallory
Mm-hmm. Yep. And I feel like it's really interesting that you bring up the expedited peer review process because that's a place that my mind has been going. So you're saying not necessarily at this moment in time having the AI peer review on its own, but being used as a tool to expedite that process. Is that correct?
[00:28:30:07 - 00:28:46:14]
Dr. Jessica Miles
Absolutely. And I'd really double click on that tool piece. I think AI is being used in a lot of areas of publishing and research, but only humans can be accountable for the research. Only humans can be accountable for the feedback that's given to researchers and authors.
[00:28:47:15 - 00:29:02:12]
Dr. Jessica Miles
And so really, AI should be a tool. AI can't author a paper. AI shouldn't author a peer review. And I think for me, it comes back to that accountability step because these tools can't be accountable for outcomes. Only humans can be accountable for outcomes.
[00:29:03:14 - 00:29:06:24]
Mallory
I love that. That's kind of a little sound bite. We're going to have to repeat throughout the episode.
[00:29:08:02 - 00:29:30:18]
Mallory
On the back end, which we've discussed at length so far, but if the metrics that we've all relied on, page views, downloads, time on page, are kind of getting muddied because we're having AI look at the content or summarize it instead of a human, what do you think publishers or leaders of associations listening to this podcast, what should be actually measured in terms of how content is performing?
[00:29:31:24 - 00:29:39:00]
Dr. Jessica Miles
No, that is like the million dollar question. I think that's one place where we're still kind of in the messy middle.
[00:29:40:12 - 00:30:50:02]
Dr. Jessica Miles
We've talked about, you talked about cloud fair and paper crawl and all of these developments point to a situation where there are tools and platforms available to distinguish between human use and bot use. I would say that's kind of the foundation right now. If you take nothing else from this episode, you certainly think about how are you measuring bona fide human use versus bot use. There are quite a lot of tools beyond cloud fair. There are a lot of providers that can give that information. In terms of how you use that information, I think there isn't really industry consensus. And I think it also depends in terms of whether or not the goal is maximizing use for research communities, in which case you might be looking at use through specialized apps or in response to specialized questions or whether it's about thinking about, okay, we see that our content is being picked up on these frontier models that are being used broadly across society. It's interesting when you look at the sites that are most frequently used in outputs,
[00:30:51:05 - 00:31:55:02]
Dr. Jessica Miles
PubMed, which is a repository of publicly funded science from the NIH rises within the top 10. I think it's probably top five because so many people are turning to these chat bots for scientific and medical questions. So one exciting opportunity is for societies to shape that information exchange more directly. The question is how, because there are so many kind of moving parts. So I think in the future, being able to more directly measure these different segments in terms of, I mentioned kind of researchers, policymakers, educators, and general public through these different AI tools, I think that would be really powerful because that's something that publishers generally have not been able to measure with a lot of granularity. But right, all societies, the mission is not only serving science, but serving society. And so getting a better sense of that second piece, serving society. I think we're not there yet, but that's where I see the future really heading.
[00:31:56:10 - 00:32:44:12]
Mallory
And to your point, especially, I feel like associations at large, this is anecdotal, but I would say are probably often cited when it comes to frontier large language models because they have some of the most trusted vetted repository of content in their respective trade or profession. However, it makes me think just because a piece of information is cited, right, doesn't necessarily mean someone might take that information from chat GPT, but they may not click through. Oh, this came from the Association of Veterinarians. Or that's not what it's called, but there may not click through to the Association website. And I'm sure you experienced the same issue when scientific research is being cited, someone might take that information or the summary of it and not necessarily click through and go to that journal. So that's that's something we're gonna have to keep an eye on for sure.
[00:32:44:12 - 00:33:29:02]
Dr. Jessica Miles
Absolutely. And that's why measuring all of these different inputs in aggregate is so important because it used to be you could just rely on full text views or downloads, but because that's becoming an increasingly smaller piece of the pie, if you're only measuring this, it looks catastrophic. You're seeing all these maybe double digit declines in activity from Google, etc. But if you're able to open the aperture and see these other types of usage, then you can think about, oh, you can set a baseline of what that's looking like. And then from there, kind of evolve different metrics, measurements, KPIs, in terms of these different types of usage from the more traditional metrics to new ways of measuring engagement.
[00:33:30:13 - 00:33:53:13]
Mallory
So with the conversation around AI, writing research, reviewing it, and reading it, I'm curious on your take now and kind of for the next few years, where do you see the integral human place in that whole process? Where do you think humans are there to stay and won't get automated out of a certain process in terms of scientific knowledge and discovery?
[00:33:53:13 - 00:36:03:21]
Dr. Jessica Miles
Well, like I said, I come back to that responsibility piece, right? Only humans can take responsibility for outcomes. And so when we think about what topics we ought to be researching, right? Where are the unanswered questions in a particular field? You know, where is their greatest need, not only from a knowledge discovery standpoint, but also from, you know, a human health or the alleviation of suffering standpoint. Those are things that AI can maybe corral data on, but only a human can make the judgment of this area versus that area. I also think when it comes to helping to train the next generation of researchers and scholars, making sure that even if they're using these AI tools, they're still able to think critically and be appropriately skeptical about their outputs. You know, an AI is not maybe best place to train a human to be skeptical of its outputs. We all know that these systems have a great deal of sycophancy built in, right? And so really relying on people who have done their training long before these AI tools were in widespread use to help educate and train, you know, the next generation and a generation coming up. That will be the responsibility of humans. And also thinking about these tools, I mean, I think we talk less about bias in AI systems than we did when generated AI was first coming out. And that's a little bit of a shame because there's still a lot of bias baked into, you know, how these systems are trained. There's still a lack of transparency when it comes to how these systems operate. And so humans are best place to kind of keep the impact of those dynamics in mind when not only building these systems, but also relying on these tools for outputs. So I think having that perspective, being able to evaluate things critically, thinking about, okay, what am I not seeing? Thinking about how can things be different than how they are right now, either with the tools or other research itself. I think those are things that I'm hopeful will remain, you know, the domain of humans in the next few years and hopefully for the years to come.
[00:36:03:21 - 00:36:53:06]
Mallory
I'm very hopeful too. We'll have a problem certainly if humans are not involved in those processes. So responsibility, the education piece I love, I wasn't expecting you to say that, but that's obviously a primary role of associations in general to educate and empower their community. So keeping that inherently human, I think is a great idea. And then also that interpretability piece, making sure we understand why these AI models are biased, what went into training them, and then hopefully providing them with more quality training content in the future, which is another area associations can play a role. I'm curious if you were advising a content heavy organization, nonprofit association on what to do in the next 12 months. This is their first engagement with you, Dr. Miles. They sit down and they say, all right, what do we do in the next 12 months? Where would you start? What would you say to them?
[00:36:54:07 - 00:38:02:24]
Dr. Jessica Miles
So when I'm working with societies, I always go back to the mission because I think unfortunately sometimes there's this impulse to say, we just have to use AI, right? And the focus is on AI and not AI as a tool. And so my first question is, what are you trying to do vis-a-vis your mission? How will whatever you're contemplating enhance your mission? And if the answer is, oh, we just want to use AI, then maybe you need to go back to the drawing table a little bit. But oftentimes when you ask it that way, you get people to surface, okay, here are the things that we're thinking about in terms of the mission, in terms of our members. And this is where we might apply AI. And that's where you can start to have these productive conversations about pilots or technologies or approaches. And then you can start to sort of frame, okay, where are the guardrails? What is it that you're absolutely... An outcome that you absolutely have to guarantee or something that you are absolutely not comfortable with? And then from there, kind of knowing outcomes, guardrails, you can start to shape strategy, activity, and pilots. So I sort of see that as kind of a virtuous circle, kind of starting with the mission and then going from there.
[00:38:02:24 - 00:38:21:00]
Mallory
I think that's a very smart way to look at it as optimistic as we tend to be about artificial intelligence on this podcast, always tying it back to business objectives. So not AI for AI's sake, but what are you trying to do in the next one, three, five years? And then how can AI amplify that and hopefully better serve your members?
[00:38:22:04 - 00:38:22:09]
Mallory
Absolutely.
[00:38:24:05 - 00:38:31:18]
Mallory
Where can people keep up with you if people are interested in finding out more about the informed frontier or more about the work that you are doing?
[00:38:32:23 - 00:38:59:13]
Dr. Jessica Miles
You can always visit our website, www.theinformedfrontier.com. You can also put in my name, just the miles, and you'll be redirected to the site. You can follow us on LinkedIn. And of course, I'm hopeful that after this conversation, you'll check out season one of the informed frontier podcast, where we look into some of these issues in a little bit more depth. You can get that on Apple, Spotify, or wherever you get your podcasts.
[00:38:59:13 - 00:39:19:20]
Mallory
I highly recommend the podcast. I listened to episode one. You did a fantastic job with it. I'm sure I'll keep listening to the rest of the episodes because I actually learned a lot in preparation for this episode. I'm curious, is there anything that we have not covered in these short 40-ish minutes that you think needs to be said when it comes to AI and publishing?
[00:39:20:22 - 00:40:01:11]
Dr. Jessica Miles
I would maybe double click on the idea of the community that society has served. And when you're thinking about these considerations of what to do, how to engage with the tools, really making sure that you are working in lockstep with your communities. I've seen some societies maybe run into challenges where they're sort of the zeal to use these tools, maybe exceeded out of their communities. So making sure you're neither too far ahead or too far behind your core constituents. So really thinking about what you'd like to do and figuring out, okay, how do I communicate this, or keep the lines of communication with the people that I'm serving open?
[00:40:03:08 - 00:40:04:09]
Mallory
Absolutely. Dr.
[00:40:04:09 - 00:40:09:22]
(Music Playing)
[00:40:20:15 - 00:40:37:14]
Mallory
Thanks for tuning into the Sidecar Sync podcast. If you want to dive deeper into anything mentioned in this episode, please check out the links in our show notes. And if you're looking for more in-depth AI education for you, your entire team, or your members, head to sidecar.ai.
[00:40:37:14 - 00:40:40:20]
(Music Playing)