October 17-21, 2022: Coalesce is back and in-person! With 100+ speakers and sessions, 3 global locations and a virtual option, it’s a week you’re not going to want to miss. Find out what’s new and what to expect from the third year of the Analytics Engineering Conference. Read now
Navigation or forgetfulness? On finding purple links in your search
We found that when you find that the top link is purple—that is, already clicked—52% of you think, “Hello, old friend.” But another 14% see that purple link with irritation and another 16% see it with amusement. There are basically two things going on here: navigational search and memory imperfections. This article will explore some of the research around both of them.
We here at Stack Overflow are flattered that we are your go to place for answers; in our 2020 Developer Survey, 90% of you said you turn to Stack Overflow when you are stuck. Not everybody goes right to the site for answers; we get that a lot of you find answers through your search engine of choice. Sometimes, you find your search turns up a purple line, which means that you’ve already searched for this problem in the past.
We found that when you find that the top link is purple—that is, already clicked—52% of you think, “Hello, old friend.” But another 14% see that purple link with irritation and another 16% see it with amusement. Maybe you don’t even remember having this problem before, much less what the solution was.
There are basically two things going on here: navigational search and memory imperfections. This article will explore some of the research around both of them.
In search of old friends
Early web search engine Altavista discovered that there were three primary types of search queries: informational, transactional, and navigational. Informational searches are when I want to know something, like the number of quarts in a gallon or the names of all the monsters in the original Destroy All Monsters. I’m looking for something new; I’m going to click on the link that looks like it best addresses my concern.
Transactional searches are where the user expects to get to a site to do something else. Buying movie tickets or ordering pad thai all fall under this kind of search. You’re not looking for a specific site from which to order these things, so any will do so long as it gets you sweet, sweet Thai food.
Navigational queries are intended to reach a specific site. Once search became automatically part of the address bar, it often was easier to search for the name of the site than enter the URL. I have absolutely typed “stack overflow” into a search bar and hit return, and I’m sure I’m not alone. Some estimate that 10% of all searches are navigational.
But some of these navigational searches are about returning to a page that you initially found via search. Perhaps it’s an excellent answer to a programming question that you found in a quick search and want to find again. You could enter the same search terms and look for the first purple link. Ah, hello, old friend, this is the exact answer I was looking for.
This is pretty common for sites with a lot of content, especially user-generated content, like Stack Overflow. Designing a navigation system that can bring you to a specific page without using search or bookmarks is incredibly hard.
The resistance of memory
There’s another pair of reactions to searching and finding a purple link: irritation and amusement (there’s also 18% who feel indifference strongly enough to select it on a survey). These seem to indicate that the person searching forgot that they even knew the solution previously. It may be that they didn’t even know that they had this problem in the past. This sort of forgetting—if even the information made it into memory in the first place—is extremely common.
Our memories are more flawed and imperfect than we would like to believe. In the book, The Seven Sins of Memory, the William R. Kenan, Jr. Professor of Psychology at Harvard, Daniel Schacter details seven ways that our memories fail us. For our inquiry into the case of the purple links, we’re interested in just two: Transience and absent-mindedness.
Transience refers to memories that just don’t stick. This happens mostly with new information that you’re trying to remember. Consider coming across a new function, keyword, or property. You read the docs, think of interesting applications for it, then the next time that you need to use it, you’ve forgotten what it’s called.
As early as 1885, German psychologist Hermann Ebbinghaus ran experiments where he memorized strings of nonsense syllables, then tested himself on how much he could recall. Through repeated memorization and testing, he determined a mathematical curve that describes how well something is retained over time. This curve has been refined by further research, but the basic idea remains the same: we forget new information unless it’s reinforced. Two studies in the late 50s found that subjects forget three nonsense syllables in as little as twenty seconds!
On top of that, the routine of work can help us forget the individual actions that we go through in a given day. A lot of daily programming happens within a work routine, which over time becomes just a part of the regular routine. A study in the 1950s asked workers to define their daily routine. Then, when asked to describe the previous day and a day from the previous week, the study found that workers remembered specific details of the previous workday, but seven days later, they only remembered tasks that fit into their daily routine. The more complicated the story they told about their average day, the more they remembered about any given day.
Most of the time, we search for answers during the workday, at least for professional programmers. The search is triggered by a problem, and unless we have “solving problems” as part of our regular workday story, this search is going to be unusual and likely forgotten soon after.
Sometimes, the issue is an attentional problem, not a recall problem; that’s related to the other memory sin involved in this: absent mindedness. Where transience described above prevents a memory from staying encoded, absent mindedness can prevent the memory from being encoded in the first place. Think of the times when you’ve lost your keys and you need to find them. You don’t remember where you put them. Even when you find your keys on top of the refrigerator, you have no idea why you put them there.
There’s a strong link between giving something your attention and being able to remember or recall it. For a while, attention was thought of like a movie; you watched the world and paid attention to everything in front of you. But then researchers found that we only saw—much less remembered—what we gave our attention to. It’s called inattentional blindness. This video is my favorite example of that.
The classic metaphor for attention is that it’s like a spotlight. You may be watching a movie, but you only pay attention to the things covered by the spotlight. Sometimes, you can pay a lot of attention to a single spot, other times you expand or split your spotlight and pay a little attention to a larger number of things. But you generally have a limited amount of attention that you can focus your spotlight on at any given time.
At work, you have tons of items demanding your attention at the same time. You’re writing code while fielding questions in your chat program, estimating story points for multiple tickets, and trying to prep for your next meeting. If you find a solution to a smaller problem, it can get lost, as it’s not part of the larger items that you’re focusing on. With so many demands on your attention, your memory can suffer. Research has shown that heavy multitaskers may have worse memory overall.
Sometimes that multitasking is just rapid task switching and none of the tasks gets full attention. Other times, you’re doing work that is so deeply in your wheelhouse, something you’ve done every day and comes almost automatically, that you can do it and another task without either suffering. But the downside is that you may find that you’ve passed a chunk of time in an automated haze and remember none of it.
There’s another draw on our attention at work: stress and anxiety. If you have a culture of constant deadlines, unsupportive or vindictive management, or even an anxiety disorder, your memory can suffer even more. You find a solution to an immediate problem and never encode it as a memory; you’ve got more pressing, longer terms worries occupying your attention.
There are simple things that you can do to make a solution stick a little better for the next time you run across the problem. Any time that you deliberately attend to something helps, especially if you do so in new ways. Something as simple as chatting with a co-worker for ten minutes about the issue and solution can help. You can come up with elaborate stories to create a richer memory (or tuck them away in your memory palace). That’s because memories get stronger through more associations.
In fact, the more exposure you give yourself to a concept, the better it sticks. But repeated exposure isn’t enough, you need to include more difficult study tasks, like recalling the stored information. You could use a flashcard program like Anki to beat back the effects of the forgetting curve. Tap into your visual memory and draw a diagram; even writing it down may help.
But don’t worry if you still find yourself forgetting some things. Forgetting is actually an essential part of any intelligent organism’s mental health. Like you do with your hard drive, your brain clears out any unused memories to make room for new information. That’s because the brain isn’t meant to retain information; it’s meant to make good decisions. If the memory of what you did to resolve an error isn’t going to help you make a good decision, pow, it’s out of there.
There’s also some concern that we are outsourcing our memory to technology. A 2013 study found that subjects who were asked to remember certain objects in a museum tour and photograph some of them actually had worse recall of the objects that they photographed. But this forgetting may be efficient as well; if you have a digital copy of the memory (or solution to a problem), why should your brain waste resources on storing it as well? You could just remember the path to it—the search phrase—and use that. Et voila, navigational search.
Memory and attention are places of active research right now. If you’re interested in seeing what our psychology Stack Exchange has to say about it, this question about the neurobiology of it is a good start.
If you’re willing to wait a little bit, you can get insight into that research in a book form. I reached out to Schacter and it turns out he’s updating the book. “When I wrote The Seven Sins of Memory two decades ago, it was already clear that memory lapses and errors have a significant and often surprising impact on everyday life,” he said. “I’m updating the book for a revised edition to appear in 2021, and it has been an interesting task to document various more recent real-world impacts of the seven sins, as well as our progress in understanding them.”
Whether you come upon a purple intentionally or unexpectedly, amused or enraged, we can only hope that it contained the answer that you needed. Solve problems, build solutions.
Creating shared meaning, together
At Coalesce 2020, we defined analytics engineering as a Community. We unpacked data transformation, Kimball, software engineering best practices, and how those all connect to the modern data analyst and data team. We built the foundation for a conference for data practitioners, by data practitioners.
At Coalesce 2021, we learned (along with over 7000 practitioners) that data work, as much as it’s about the code and technology that drives it, is about people. It’s about how we collaborate as data practitioners to solve interesting technical problems; how we build empathy and relationships with our stakeholders; and how we create tools and systems that are built for the people using them. And how we do these things together as a Community.
“A must-attend for my entire team in 2021. Best lineup of talks of any data conference and great energy. It’s real conversations about data topics that matter.”
At Coalesce 2022, we’re building upon those learnings to focus on understanding how data practitioners are moving beyond their teams and driving tangible value for their organizations. This 5-day in-person and virtual experience kicks off on October 17th, with the first of more than 100 thoughtfully curated sessions designed to make us rethink how we work within, and beyond our data teams.
Beyond learning, Coalesce is about connecting with people. This year, analytics engineers, data engineers, data analysts, data scientists, executives, and many others are tuning in from 78+ countries and 2300+ companies. Folks will be able to interact online, in-person, or a mix of the two through the Online, New Orleans, and Online+ location offerings.
Hands will be shaked, memes will be shared, things will be learned, and connections will be made. We can’t wait to share what’s happening at Coalesce 2022, and we hope we can see you there.
Give me the good stuff! The TL;DR of Coalesce 2022
For five days, analytics engineering practitioners will be gathering together to focus on learning and sharing the skills, tools, and methods to produce higher leverage and higher impact data work. This year, Coalesce is offering in-person, online, and virtual versions of the conference:
- Coalesce New Orleans (Oct. 17-21, 2022): Join us in NOLA! Live workshops, sessions, a hackathon…we’ll help you take advantage of rare face-to-face time with friends new and old.
- Coalesce Online+ London (Oct. 20, 2022): Calling all folks in EMEA! A special one-day in-person supplement to the online Coalesce experience.
- Coalesce Online+ Sydney (Oct. 21, 2022): For our friends in APAC, a one-day in-person companion to the online Coalesce experience.
- Coalesce Online (Oct. 17-21, 2022): The livestream worth clearing your calendar for. No matter where you are in the world—get the Coalesce experience from the comfort of your home!
At the heart of it all: Practitioner and community-driven knowledge sharing
“Coalesce presentations are just all of us, trying to save each other time that we have spent learning hard things. -Erin Vaughan (dbt Labs)”
At Coalesce 2022, there will be over 100 ecosystem and Community sessions covering a spectrum of concepts, from mind-boggling SQL to the philosophical questions keeping data teams up at night. Below are some of the session themes shaping this year’s Coalesce experience:
- ⚙️Technical Impact: The nitty-gritty technical skills and knowledge that will help you level up your data skills and data stack.
- Mindset Shift: New tools invite new modes of thinking and problem solving that challenge our existing ways of working. Keep up with the changes to learn which shifts are paying off and why.
- ️ Navigate Your Career: Strategies and insights for analytics practitioners who are looking for ways to increase their impact.
- ⚡️ Leading Teams: From executive leaders to team leads, learn how individuals and teams are positioning themselves for impact within their organizations.
- Just Getting Started: Insider perspectives designed to orient and welcome those who are new to analytics engineering.
New to dbt and analytics engineering? A regular commenter in #advice-dbt-for-power-users? Need some help identifying new tools for your data stack? Practitioners are bringing their exciting and novel perspectives and experiences to create sessions for a variety of skill sets, interests, and needs. Below is a peek at some of the sessions:
- ❄️dbt Labs + Snowflake: Why SQL and Python go perfectly well together
- ️dbt Labs and Databricks: Best practices and future roadmap
- dbt_project_evaluator with Grace Goheen (dbt Labs)
- What does analytics engineering have to do with product experimentation? with Adam Stone (Netlify)
- Empowering pythonistas with dbt and Snowpark with Eda Johnson (Snowflake)
- ❣️Empathy building in data work with Lorena Vazquez (Wonder)
- Outgrowing a single `dbt run` with Prratek Ramchandani (Vox Media)
- ✍️How not to be a terrible writer with Justin Gage (Amplify Partners)
- Sculpting data for machine learning with Jigyasa Grover and Rishabh Misra (Twitter)
All the things that make Coalesce, well, Coalesce
Sticking to what we know while expanding our horizons.
Coalesce starts in Slack
This year, while Coalesce attendees have the opportunity to choose between live or virtual events, all sessions start in the dbt Community Slack. Each session is provisioned with its own Slack channel where session attendees can bring the memes, excitement, and knowledge to the shared space.
All sessions have dedicated moderators and guidelines to help you navigate through the comments and experience. Folks are encouraged to bring the energy, questions, and emojis to the chat. Team members from the speaker’s organization will also be directed to the Slack channel after their session ends to help answer questions and engage in thoughtful debate.
After sessions are over, recordings will be available so you can watch them on the schedule that works best for you…and then hop into the Slack channel to join in the conversation and catch the tea
Whether you’re joining us in London, Sydney, NOLA, or from your living room couch, we look forward to your engagement and participation in Slack!
New product feature announcements and launches
Come join the dbt Labs product team as they release and demo some of the newest product features:
- The Semantic Layer: An unearthed level of uniformity and governance brought to analytics and a humble reminder that data work is, well, for the people.
- Python Models in dbt: The first step to building a more complex DAG and closing the gap between analytics engineers, traditional data teams, and ML/DS.
- IDE Improvements: Reigniting the connection between analysts and the analytics engineering workflow.
Can’t wait until October to see these product features? Check out the docs on the Semantic Layer, Python models in dbt, and IDE improvements.
Hackathon, workshops, certifications, oh my!
New to Coalesce this year, attendees can expect engaging and informative virtual and in-person learning events happening throughout the week.
- Hackathon: Taking place on the first day of Coalesce New Orleans, join the dbt Labs engineering and product teams as we work through contributing to open source analytics engineering work, together.
- Partner Day: Day 1 (1:30 pm – 5pm) at Coalesce New Orleans will have programming for dbt Labs Technology and Consulting Program partners. Partner Day will include in-person breakout sessions, networking events, and other activities. For more information on Partner Day, check out the resources on the Coalesce website or reach out to your partner manager.
- ⚒️ Workshops: dbt Labs team members and Coalesce sponsors will be hosting hands-on and interactive learning workshops throughout the week for New Orleans attendees that will be livestreamed for Online folks.
- ️ Trainings and dbt Certification: The dbt Labs team will be hosting several live trainings for dbt users of all skill levels. For practitioners ready for the challenge, we’ll also be offering an in-person exam for dbt Certification. Databricks and Snowflake will also be hosting training sessions on Day 1 of the conference.
Mingling in the Big Easy
Attendees at the Coalesce New Orleans can expect a variety of jazzy in-person events. After all, we’re going to be in New Orleans!
- A second line parade to Champion Square
- Sponsor happy hours, networking events, and parties
- An activation hall that takes the typical conference booth to a more meaningful and personal level
- A closing party at Mardi Gras World!
- Not on the official agenda, but…1:1 face time with that Slack name you’ve always wanted to connect with and learn from
Folks at the Coalesce Online+ locations in Sydney and London can also expect in-person networking sessions and live executive keynotes.
