Tag Archives: firefox

Using Data to Make Firefox Better: A mini-case study for your organization

I love Mozilla. Any reader of this blog knows it. I believe in its mission, I find the organization totally fascinating and its processes engrossing. So much so I spend a lot of time thinking about it – and hopefully, finding ways to contribute.

I’m also a big believer in data. I believe in the power of evidence-based public policy (hence my passion about the long-form census) and in the ability of data to help organizations develop better products, and people make smarter decisions.

Happily, a few months ago I was able to merge these two passions: analyzing data in an effort to help Mozilla understand how to improve Firefox. It was fun. But more importantly, the process says a lot about the potential for innovation open to organizations that cultivate an engaged user community.

So what happened?

In November 2010, Mozilla launched a visualization competition that asked: How do People Use Firefox? As part of the competition, they shared anonymous data collected from Test Pilot users (people who agreed to share anonymous usage data with Mozilla). Working with my friend (and quant genius) Diederik Van Liere, we analyzed the impact of add-on memory consumption on browser performance to find out which add-ons use the most memory and thus are most likely slowing down the browser (and frustrating users!). (You can read about our submission here).

But doing the analysis wasn’t enough. We wanted Mozilla engineers to know we thought that users should be shown the results – so they could make more informed choices about which add-ons they download. Our hope was to put pressure on add-on developers to make sure they weren’t ruining Firefox for their users. To do that we visualized the data by making a mock up of their website – with our data inserted.

FF-memory-visualizations2.001

For our efforts, we won an honourable mention. But winning a prize is far, far less cool than actually changing behaviour or encouraging an actual change. So last week, during a trip to Mozilla’s offices in Mountain View, I was thrilled when one of the engineers pointed out that the add-on site now has a page where they list add-ons that most slow down Firefox’s start up time.

Slow-Performing-Add-ons-Add-ons-for-Firefox_1310962746129

(Sidebar: Anyone else find it ironic that “FastestFox: Browse Faster” is #5?)

This is awesome! Better still, in April, Mozilla launched an add-on performance improvement initiative to help reduce the negative impact add-ons can have on Firefox. I have no idea if our submission to the visualization competition helped kick-start this project; I’m sure there were many smart people at Mozilla already thinking about this. Maybe it was already underway? But I like to believe our ideas helped push their thinking – or, at least, validated some of their ideas. And of course, I hope it continues to. I still believe that the above-cited data shouldn’t be hidden on a webpage well off the beaten path, but should be located right next to every add-on. That’s the best way to create the right feedback loops, and is in line with Mozilla’s manifesto – empowering users.

Some lessons (for Mozilla, companies, non-profits and governments)

First lesson. Innovation comes from everywhere. So why aren’t you tapping into it? Diederik and I are all too happy to dedicate some cycles to thinking about ways to make Firefox better. If you run an organization that has a community of interested people larger than your employee base (I’m looking at you, governments), why aren’t you finding targeted ways to engage them, not in endless brainstorming exercises, but in innovation challenges?

Second, get strategic about using data. A lot of people (including myself) talk about open data. Open data is good. But it can’t hurt to be strategic about it as well. I tried to argue for this in the government and healthcare space with this blog post. Data-driven decisions can be made in lots of places; what you need to ask yourself is: What data are you collecting about your product and processes? What, of that data, could you share, to empower your employees, users, suppliers, customers, whoever, to make better decisions? My sense is that the companies (and governments) of the future are going to be those that react both quickly and intelligently to emerging challenges and opportunities. One key to being competitive will be to have better data to inform decisions. (Again, this is the same reason why, over the next two decades, you can expect my country to start making worse and worse decisions about social policy and the economy – they simply won’t know what is going on).

Third, if you are going to share, get a data portal. In fact, Mozilla needs an open data portal (there is a blog post that is coming). Mozilla has always relied on volunteer contributors to help write Firefox and submit patches to bugs. The same is true for analyzing its products and processes. An open data portal would enable more people to help find ways to keep Firefox competitive. Of course, this is also true for governments and non-profits (to help find efficiencies and new services) and for companies.

Finally, reward good behaviour. If contributors submit something you end up using… let them know! Maybe the idea Diederik and I submitted never informed anything the add-on group was doing; maybe it did. But if it did… why not let us know? We are so pumped about the work they are doing, we’d love to hear more about it. Finding out by accident seems like a lost opportunity to engage interested stakeholders. Moreover, back at the time, Diederik was thinking about his next steps – now he works for the Wikimedia Foundation. But it made me realize how an innovation challenge could be a great way to spot talent.

Developing Community Management Metrics and Tools for Mozilla

Background – how we got here

Over the past few years I’ve spent a great deal of time thinking about how we can improve both the efficiency of open source communities and contributors experience. Indeed, this was the focus, in part, of my talk at the Mozilla Summit last summer. For some years Diederik Van Liere – now with the Wikimedia foundation’s metrics team – and I have played with Bugzilla data a great deal to see if we could extract useful information from it. This led us to engaging closely with some members of the Mozilla Thunderbird team – in particular Dan Mosedale who immediately saw its potential and became a collaborator. Then, in November, we connected with Daniel Einspanjer of Mozilla Metrics and began to imagine ways to share data that could create opportunities to improve the participation experience.

Yesterday, thank’s to some amazing work on the part of the Mozilla Metrics team (listed at bottom of the post), we started sharing some of work at the Mozilla all hands. Specifically, Daniel demoed the first of a group of dashboards that describe what is going on in the Mozilla community, and that we hope, can help enable better community management. While these dashboards deal with the Mozilla community in particular I nonetheless hope they will be of interest to a number of open source communities more generally. (presently the link is only available to Mozilla staffers until the dashboard goes through security review – see more below, along with screen shots – you can see a screencast here).

Why – the contributor experience is a key driver for success of open source projects

My own feeling is that within the Mozilla community the products, like Firefox, evolve quickly, but the process by which people work together tends to evolve more slowly. This is a problem. If Mozilla cannot evolve and adopt new approaches with sufficient speed then potential and current contributors may go where the experience is better and, over time, the innovation and release cycle could itself cease to be competitive.

This task is made all the more complicated since Mozilla’s ability to fulfill its mission and compete against larger, better funded competitors depends on its capacity to tap into a large pool of social capital – a corps of paid and unpaid coders whose creativity can foster new features and ideas. Competing at this level requires Mozilla to provide processes and tools that can effectively harness and coordinate that energy at minimal cost to both contributors and the organization.

As I discussed in my Mozilla Summit talk on Community Management, processes that limit the size or potential of our community limit Mozilla. Conversely, making it easier for people to cooperate, collaborate, experiment and play enhances the community’s capacity. Consequently, open source projects should – in my opinion – constantly be looking to reduce or eliminate transactions costs and barriers to cooperation. A good example of this is how Github showed that forking can be a positive social contribution. Yes it made managing the code base easier, but what it really did was empower people. It took something everyone thought would kill open source projects – forking – and made it a powerful tool of experimentation and play.

How – Using data to enable better contributor experience

Unfortunately, it is often hard to quantitatively asses how effectively an open source community manages itself. Our goal is to change that. The hope is that these dashboards – and the data that underlies them – will provide contributors with an enhanced situational awareness of the community so they could improve not just the code base, but the community and its processes. If we can help instigate a faster pace of innovation of change in the processes of Mozilla, then I think this will both make it easier to improve the contributor experience and increase the pace of innovation and change in the software. That’s the hope.

That said, this first effort is a relatively conservative one. We wanted to create a dashboard that would allow us to identify some broader trends in the Mozilla Community, as well as provide tangible, useful data to Module Owners – particularly around identifying contributors who may be participating less frequently.


This dashboard is primarily designed to serve two purposes. First is to showcase what dashboards could be with the hope of inspiring the Mozilla community members to use it and, more importantly, to inspire them to build their own. The second reason was to provide module owners with a reliable tool with which to more effective manage their part of the community.  So what are some of the ways I hope this dashboard might be helpful? One important feature is the ability to sort contributors by staff or volunteer. An open source communities volunteer contributors should be a treasured resource. One nice things about this dashboard is that you can not only see just volunteers, but you can get a quick sense of those who haven’t submitted a patch in a while.

In the picture below I de-selected all Mozilla employees so that we are only looking at volunteer contributors. Using this view we can see who are volunteers who are starting to participate less – note the red circle marked “everything okay?” A good community manager might send these people an email asking if everything is okay. Maybe they are moving on, or maybe they just had a baby (and so are busy with a totally different type of patch – diapers), but maybe they had a bad experience and are frustrated, or a bunch of code is stuck in review. These are things we would want to know, and know quickly, as losing these contributors would be bad. In addition, we can also see who are the emerging power contributors – they might be people we want to mentor, or connect with mentors in order to solidify their positive association with our community and speed up their development. In my view, this should be core responsibilities of community managers and this dashboard makes it much easier to execute on these opportunities.

main-dasboard-notes
Below you can see how zooming in more closely allows you to see trends for contributors over time. Again, sometimes large changes or shifts are for reasons we know of (they were working on features for a big release and its shipped) but where we don’t know the reasons maybe we should pick up the phone or email this person to check to see if everything is okay.

user-dashboard-notes

Again, if this contributor had a negative experience and was drifting away from the community – wouldn’t we want to know before they silently disappeared and moved on? This is in part the goal.

Some of you may also like the fact that you can dive a little deeper by clicking on a user to see what specific patches that user has worked on (see below).

User-deep-dive1

Again, these are early days. My hope is that other dashboards will provide still more windows into the community and its processes so as to show us where there are bottlenecks and high transaction costs.

Some of the features we’d like to add to this or other dashboards include:

  • a code-review dashboard that would show how long contributors have been waiting for code-review, and how long before their patches get pushed. This could be a powerful way to identify how to streamline processes and make the experience of participating in open source communities better for users.
  • a semantic analysis of bugzilla discussion threads. This could allow us to flag threads that have become unwieldy or where people are behaving inappropriately so that module owners can better moderate or problem solve them
  • a dashboard that, based on your past bugs and some basic attributes (e.g. skillsets) informs newbies and experienced contributors which outstanding bugs could most use their expertise
  • Ultimately I’d like to see at least 3 core dashboards – one for contributors, one for module owners and one for overall projects – emerge and, as well as user generated dashboards developed using Mozilla metrics data.
  • Access to all the data in Bugzilla so the contributors, module owners, researchers and others can build their own dashboards – they know what they need better than we do

What’s Next – How Do I Get To Access it and how can I contribute

Great questions.

At the moment the dashboard is going through security review which it must complete before being accessible. Our hope is that this will be complete by the end of Q2 (June).

More importantly, we’d love to hear from contributors, developers and other interested users. We have a standing weekly call every other Friday at 9am PST where we discuss development issues with this and the forthcoming code-review dashboard, contributors needs and wanted features, as well as use cases. If you are interested in participating on these calls please either let me know, or join the Mozilla Community Metrics Google group.

Again, a huge shout out is deserved by Daniel Einspanjer and the Mozilla Metrics team. Here is a list of contributors both so people know who they are but also in case anyone has question about specific aspects of the dashboard:
Pedro Alves – Team Lead
Paula Clemente – Dashboard implementor
Nuno Moreira – UX designer
Maria Roldan – Data Extraction
Nelson Sousa – Dashboard implementor

Visualizing Firefox Plugins Memory Consumption

A few months ago the Mozilla Labs and the Metrics Team, together with the growing Mozilla Research initiative, launched an Open Data Visualization Competition.

Using data collected from Test Pilot users (people who agreed to share anonymous usage data with Mozilla and test pilot new features) Mozilla asked its community to think of creative visual answers to the question: “How do people use Firefox?”

As an open data geek and Mozilla supporter the temptation to try to do something was too great. So I teamed up with my old data partner Diederik Van Liere and we set out to create a visualization. Our goals were simple:

  • have fun
  • focus on something interesting
  • create something that would be useful to Firefox developers and/or users
  • advance the cause for creating a Firefox open data portal

What follows is the result.

It turns out that – in our minds – the most interesting data set revolved around plugin memory consumption. Sure this sounds boring… but plugins (like Adobe reader, Quicktime or Flash) or are an important part of the browser experience – with them we engage in a larger, richer and more diverse set of content.  Plugins, however, also impact memory consumption and, consequently, browser performance. Indeed, some plugins can really slow down Firefox (or any browser). If consumers had a better idea of how much performance would be impacted they might be more selective about which plugins they download, and developers might be more aggressive in trying to make their plugins more efficient.

Presently, if you run Firefox you can go to the Plugin Check page to see if your plugins are up to date. We thought: Wouldn’t it be great if that page ALSO showed you memory consumption rates? Maybe something like this (note the Memory Consumption column, it doesn’t exist on the real webpage, and you can see a larger version of this image here):

Firefox data visualization v2

Please understand (and we are quite proud of this). All of the data in this mockup is real. Memory consumptions are estimates we derived by analyzing the Test Pilot data.

How, you might ask did we (Diederik) do that?

GEEK OUT EXPLANATION: Well, we (Diederik) built a dataset of about 25,000 different testpilot users and parsed the data to see which plugins were installed and how much memory was consumed around the time of initialization. This data was analyzed using ordinary least squares regression where the dependent variable is memory consumption and the different plugins are the explanatory variables. We only included results that are highly significant.

The following table shows our total results (you can download a bigger version here).

Plugin_memory_consumption_chart v2

Clearly, not all plugins are created equal.

Our point here isn’t that we have created the definitive way of assessing plugin impact on the browser, our point is that creating a solid methodology for doing so is likely witihin Mozilla’s grasp. More importantly, doing this could help improve the browsing experience. Indeed, it would probably be even wiser to do something like this for Add-ons, which is where I’m guessing the real lag time around the browsing experience is created.

Also, with such a small data set we were only able to calculate the memory usage for a limited number of plugins and generally those that are more obscure. Our methodology required having several data points from people who are and who aren’t using a given plugin and so with many popular plugins we didn’t have enough data from people who weren’t using it… a problem however, that would likely be easily solved with access to more data.

Finally, I hope this contest and our submission helps make the case for why Mozilla needs an open data portal. Mozilla collects and incredible amount of data of which it does not have the resources to analyze internally. Making it available to the community would do to data what Mozilla has done to code – enable others create value that could affect the product and help advance the open web. I had a great meeting earlier this week with a number of the Mozilla people about this issue, I hope that we can continue to make progress.

Canadian Governments: How to waste millions online ($30M and counting)

Back from DC and Toronto I’m feeling recharged and reinvigorated. The Gov 2.0 expo was fantastic, it was great to meet colleagues from around the world in person. The FCM AGM was equally exciting with a great turnout for our session on Government 2.0 and lots of engagement from the attendees.

So, now that I’m in a good mood, it’s only natural that I’m suddenly burning up about some awesomely poor decisions being made at the provincial level and that may also may be in the process of being made at the federal level.

Last year at the BC Chapter of the Municipal Information Systems Association conference I stumbled, by chance, into a session run by the British Columbia government about a single login system it was creating for government website. So I get that this sounds mundane but check this out: it would means that if you live in BC you’ll have a single login name and password when accessing any provincial government service. Convenient! Better still, the government was telling the municipalities that this system (still in development) could work for their websites too. So only one user name and password to access any government service in BC! It all sounds like $30 million (the number I think they quoted) well spent.

So what could be wrong with this…?

How about the fact that such a system already exists. For free.

Yes, OpenID, is a system that has been created to do just this. It’s free and licensed for use by anyone. Better still, it’s been adopted by a number of small institutions such as Google, Yahoo, AOL, PayPal, and Verisign and… none other than the US government which recently began a pilot adoption of it.

So let me ask you: Do you think the login system designed by the BC government is going to be more, or less secure that that an open source system that enjoys the support of Google, Yahoo, AOL, PayPal, Verisign and the US Government? Moreover, do we think that the security concerns these organizations have regarding their clients and citizens are less strict than those of the BC government?

I suspect not.

But that isn’t going to prevent us from sinking millions into a system that will be less secure and will costs millions more to sustain over the coming decades (since we’ll be the only ones using it… we’ll have to have uniquely trained people to sustain it!).

Of course, it gets worse. While the BC government is designing its own system, rumour has it that the Federal Government is looking into replacing Epass; it’s own aging website login system which, by the by, does not work with Firefox, a web browser used by only a quarter of all Canadians. Of course, I’m willing to bet almost anything that no one is even contemplating using OpenID. Instead, we will sink 10’s of millions of dollars (if not more…) into a system. Of course, what’s $100 million of taxpayer dollars…

Oh, and today’s my birthday! And despite the tone of this post I’m actually in a really good mood and have had a great time with friends, family and loved ones. And where will I be today…? At 30,000 ft flying to Ottawa for GovCamp Canada. Isn’t that appropriate? :)

Some More Core-Periphary Maps

Those who’ve been reading my blog for a long time may remember one of my more popular posts comparing the Firefox 3 Pledge Map (locations of downloads of Firefox 3 back in June 2008) versus Thomas Barnett’s Map (published in The Pentagon’s New Map – his blog here).

PNM%20remixed%202

firefox PNM mash up 2

A little while back a friend shared with me a new map, called The Walled World, that she’d found over at The Raw Feed (a great site, BTW) which offers a similar perspective… but with clearly delineated walls that show who is being kept out of which parts of the world.

the-walled-world-large

All three maps continue reasonate with me. The first offers us a stategic overlay. Which countries are powers/maintainers of the international system – which places are seeking to radical alter it, or cannot seem to become part of the core.

The second shows the virtual implications of that gap. Here, the gap between core and periphery is made starkly clear in technology use.

The final shows the physical manifestation of the gap. A stark reminder of the fences we build and the enormous sums of money and energy poured into keeping certain people out.

As a final note, I do think the third map is slightly misleading. As disturbing as it is, it is actually far, far too flattering to many traditional western powers as it continues to place them at the “centre.” In a world where the United States appears to be in decline this type of map makes China, Brazil, India and Russia (and even South Africa) look like non entities. Nothing could be further from the truth.

Searching The Vancouver Public Library Catalog using Amazon

A few months ago I posted about a number of civic applications I’d love to see. These are computer, iphone, blackberry applications or websites that leverage data and information shared by the government that would help make life in Vancouver a little nicer.

Recently I was interviewed on CBC’s spark about some of these ideas that have come to fruition because of the hard work and civic mindedness of some local hackers. Mostly, I’ve talked about Vantrash (which sends emails or tweets to remind people of their upcoming garbage day), but during the interviewed I also mentioned that Steve Tannock created a script that allows you to search the Vancouver Public Library (VPL) Catalog from the Amazon website.

Firstly – why would you want you want to use Amazon to search the VPL? Two reasons: First, it is WAY easier to find books on the Amazon site then the library site, so you can leverage Amazon’s search engine to find books (or book recommendations) at the VPL. Second, it’s a great way to keep the book budget in check!

To use the Amazon website to search the VPL catalog you need to follow these instructions:

1. You need to be using the Firefox web browser. You can download and install it for free here. It’s my favourite browser and if you use it, I’m sure it will become yours too.

2. You will need to install the greasemonkey add-on for Firefox. This is really easy to do as well! After you’ve installed Firefox, simply go here and click on install.

3. Finally, you need to download the VPL-Amazon search script from Steve Tannock’s blog here.

4. While you are at Steve’s blog, write something nice – maybe a thank you note!

5. Go to the Amazon website and search for a book. Under the book title will be a small piece of text letting you know if the VPL has the book in its catalog! (See example picture below) Update: I’m hearing from some users that the script works on the Amazon.ca site but not the Amazon.com site.

I hope this is helpful! And happy searching.

Also, for those who are more technically inclined feel free to improve on the script – fix any bugs (I’m not sure there are any) or make it better!

Amazon shot

5 Ways to get to the Next Million Mozillians

Mark Surman, Executive Director of the Mozilla Foundation has been ruminating on:

how Mozilla can actively encourage large numbers of people to participate on making the web more open and awesome.”

For a long time I’ve been a supporter of the idea that supporters of an Open Web are part of a social movement and that mobilizing these supporters could be a helpful part of a strategy for preserving and promoting the openness of the web. More importantly, I think the rise of open source, and in particular the rise of Mozilla tracks shockingly well against the structure of a social movement.

So if we are interested in increasing interest in the openness of the web and believe that recruiting the next million Mozillians can helps us accomplish that, then I think there are 3 things any strategy must do:

1. Increase the range of stake holders involved (this is part of why I write about women in open source so much) as this gives open web supporters more leverage when negotiating with those who threaten the web’s openness or who influence its development

2. Connect nebulous ideas like “security” and “openness” to tangible experiences people (users?) can relate to and to core values they believe in. (This is why Mark’s “seatbelt moment” narrative is awesome in this regard)

3. Outline actions that stakeholders and supporters can take.

So with the (not always successful) intent to focus on these 3 objectives here are five ideas I think could help us:

Idea 1: Partner with Consumer Reports and help shape the criteria by which they evaluate ISPs

One key to ensuring an open web is ensuring that people’s connection to the web is itself open. ISPs are a critical component in this ecosystem. As some observers have noted, ISPs engage in all sorts of nefarious activities such as bandwidth shaping, throttling, etc… Ensure that the net stays neutral feels like a critical part of ensuring it stays open.

One small way to address this would be make the neutrality of a network part of the evaluation criteria for Consumer Report reviews of ISPs. This would help make the openness of an ISP a competitive, would increase the profile of this problem and would engage a group of people (Consumer Report users) that are probably not generally part of the Mozilla Community.

Idea 2: Invest in an enterprise level support company for Firefox & Thunderbird.

Having a million citizens supporting Firefox and Mozilla is great, but if each of those supporters looks and acts the same then their impact is limited. Successful movements are not just large they are also diverse. This means having a range of stakeholders to help advocate for the open web. One powerful group of stakeholders are large enterprises & governments. They have money, they have clout and they have large user bases. They are also – as far as I can tell – one of the groups that Firefox has had the hardest time achieving market share with.

From my limited experience working with governments, adopting Firefox is difficult. There is no one to sign an SLA with, no dedicated support desk and no assurances problems will be escalated to the right developer within a fixed time period. Many of these challenges are highlighted by Tauvix in the comment section of this post). We could spend our time arguing about whether these issues are legitimate or if those large organizations simply need a culture shift. But such a shift will take a LONG time to materialize, if it ever does.

Finding a way to satisfy the concerns of large organizations – perhaps through a Redhat type model – might be a good way to invest Mozilla Foundation money. Not only could there be a solid return on this investment, but it could bring a number of large powerful companies and governments into the Mozilla camp. These would be important allies in the quest for an open web.

Idea 3: Promote add-ons that increase security, privacy and control in the cloud.

One reason behind Mozilla’s enormous success is that the community has always provided innovative technical solutions to policy/privacy/openness problems. Don’t like they way Microsoft is trying to shape the internet? Here, use a better browser. Don’t want to receive target advertising on website? Here, download this plug-in. Don’t want to see any advertising? Here, download this plug-in. Not sure if a website is safe? Here, use this plug-in. In short, Mozilla has allowed its software to serve as a laboratory to experiment with new and interesting ways to allow users to control their browsing experience.

While not a complete solution, it might be interesting to push the community to explore how Greasemonkey scripts, Jetpack plug-ins, or ordinary plug-ins might provide users with greater control over the cloud. For example, could a plug-in create automatic local backups of google docs on your computer? Could a Thunderbird plugin scan facebook messages and allow users a choice of mediums to respond with (say email). Fostering a “product-line” of cloud specific plug-ins that increase user control over their experience might be an interesting place to start.

Idea 4: Create and brand the idea of an openness audit

As more and more personal data ends up in servers controlled by companies, governments and non-profits there are real concerns around how secure and private this information is. Does anyone know that Google isn’t peeking at your Google docs every once in a while? Do you know if you’ll ever be able to delete your personal information from facebook?

These are legitimate questions. Outlining some guidelines around how companies manage privacy and security and then creating an audit system might be an interesting way to nudge companies towards adopting stronger standards and policies in the cloud. This might also increase public awareness and encourage a upwards spiral among competing service providers. Working with companies like KPMG and Deloitte Mozilla and others could help foster a new type of audit, one that would allow consumers to easily discriminate against cloud service providers that respect their rights, and those that don’t.

Idea 5: Let’s use that Firefox launch screen to create the next million Mozillians

At the moment, when you download and install Firefox the first website you see when you load the program congratulates you on downloading the program, tells you that you are helping keep the internet open and outlines some of Firefox’s new features. We could do more. Why not prompt people to join a “Mozillians” club where they will be kept up to date on threats and opportunities around the open web. Or maybe we should list 3 actions (with hyperlinks) they can take to increase the openness of the web (say, upgrade a friend, send a form letter to their member of congress and read an intro article on internet security?)

With maybe 300+ million people likely to download Firefox 3.5, that’s a lot of people we could be mobilizing to be more active, technically, socially and politically, around an open web.

There’s a start… I’ll keep brainstorming more ideas but in the interim, please feel free to let me know if you think any of these have real problems and/or are bunk.